Virtualized file server disaster recovery

ABSTRACT

In one embodiment, a system for managing a virtualization environment includes a set of host machines, each of which includes a hypervisor, virtual machines, and a virtual machine controller, and a virtualized file server backup system configured to identify backup data, wherein the backup data comprises data stored on the virtual disks and VFS configuration information, and the first data is identified in accordance with a backup policy, send the backup data to one or more remote sites for storage, and, in response to detection of changes in the backup data, send the changes to the remote sites in accordance with a replication policy. The backup data may be identified based on a protection domain associated with the backup policy. The data stored on the VFS may include one or more storage objects. The storage objects may include shares, groups of shares, files, or directories.

PRIORITY

This application is a continuation of U.S. application Ser. No.15/422,286 filed Feb. 1, 2017, which claims the benefit, under 35 U.S.C.§ 119(e), of U.S. Provisional Patent Application No. 62/294,984, filedFeb. 12, 2016, which are incorporated herein by reference, in theirentirety, for any purpose.

TECHNICAL FIELD

This disclosure generally relates to file servers in virtualizedenvironments.

BACKGROUND

A virtual machine (“VM”) may refer to a specific software-basedimplementation of a machine in a virtualization environment, in whichthe hardware resources of a real computer (e.g., CPU, memory, etc.) arevirtualized or transformed into the underlying support for the fullyfunctional virtual machine that can run its own operating system andapplications on the underlying physical resources just like a realcomputer.

Virtualization works by inserting a thin layer of software directly onthe computer hardware or on a host operating system. This layer ofsoftware contains a virtual machine monitor or “hypervisor” thatallocates hardware resources dynamically and transparently. Multipleoperating systems run concurrently on a single physical computer andshare hardware resources with each other. By encapsulating an entiremachine, including CPU, memory, operating system, and network devices, avirtual machine is completely compatible with most standard operatingsystems, applications, and device drivers. Most modern implementationsallow several operating systems and applications to safely run at thesame time on a single computer, with each having access to the resourcesit needs when it needs them.

Virtualization allows one to run multiple virtual machines on a singlephysical machine, with each virtual machine sharing the resources ofthat one physical computer across multiple environments. Differentvirtual machines can run different operating systems and multipleapplications on the same physical computer.

One reason for the broad adoption of virtualization in modern businessand computing environments is because of the resource utilizationadvantages provided by virtual machines. Without virtualization, if aphysical machine is limited to a single dedicated operating system, thenduring periods of inactivity by the dedicated operating system thephysical machine is not utilized to perform useful work. This iswasteful and inefficient if there are users on other physical machineswhich are currently waiting for computing resources. To address thisproblem, virtualization allows multiple VMs to share the underlyingphysical resources so that during periods of inactivity by one VM, otherVMs can take advantage of the resource availability to processworkloads. This can produce great efficiencies for the utilization ofphysical devices, and can result in reduced redundancies and betterresource cost management.

Furthermore, there are now products that can aggregate multiple physicalmachines, running virtualization environments to not only utilize theprocessing power of the physical devices to aggregate the storage of theindividual physical devices to create a logical storage pool wherein thedata may be distributed across the physical devices but appears to thevirtual machines to be part of the system that the virtual machine ishosted on. Such systems operate under the covers by using metadata,which may be distributed and replicated any number of times across thesystem, to locate the indicated data. These systems are commonlyreferred to as clustered systems, wherein the resources of the group arepooled to provide logically combined, but physically separate systems.

SUMMARY OF PARTICULAR EMBODIMENTS

In particular embodiments, a virtualized file system (VFS) disasterrecovery system may create backups of data that is stored in a VFS. Adata backup may be a copy of a VFS's stored data, e.g., copies of allthe files, folders, and metadata stored by the VFS. If data loss occurs,e.g., because of a disaster, hardware failure, or other type of failure,then the disaster recovery system may recover from the data loss byrestoring the VFS from the data backup, e.g., by creating a new VFSinstance and copying the data from the backup to the new VFS instance.The recovery system may also replicate delta changes in the storagelayer so that a data backup may be supplemented over time with morerecent data by storing changes to the backup in an incremental manner.When recovering from a failure, the recovery system may restore the mostrecent data backup and apply delta changes that are more recent that thedata backup to the restored data.

In particular embodiments, the data stored on a VFS may be securelyprotected and restored at a remote location without loss of stored data,and with a defined service level. The service level may specify that thedata is to be recovered within a defined time period, e.g., within asupported Recovery Point Objective. A Recovery Point Objective may be,for example, the maximum amount of time for which the VFS may beunavailable without irretrievably losing data. System administrators orusers may apply the replication and backup policies on the protectiondomain to configure the Recovery Point Objective, recovery sites(alternate cluster or cloud), and replication constraints such asbandwidth and schedule. The recovery system may map the VFSconfiguration between sites to provide disaster recovery of virtual fileservices across geographical locations. Particular shares may be mappedto particular remote sites.

Particular embodiments may provide a user interface to configuremappings of network, DNS-servers, active-directory, and so on betweenremote sites. The user interface may provide one-click restore of VFSfile-services on remote sites. In particular embodiments, the backupsystem may provide a granular level of protection (share or group ofshares) to configure different Recovery Point Objectives per share orper group of shares. For example, the granularity may be share-level toindicate that shares are to be backed up as separate units withconsistency preserved inside each share, or group-of-share-level toindicate that defined groups of shares are to be backed up as units withconsistency preserved inside each group.

Further details of aspects, objects, and advantages of the invention aredescribed below in the detailed description, drawings, and claims. Boththe foregoing general description and the following detailed descriptionare exemplary and explanatory, and are not intended to be limiting as tothe scope of the invention. Particular embodiments may include all,some, or none of the components, elements, features, functions,operations, or steps of the embodiments disclosed above. The subjectmatter which can be claimed comprises not only the combinations offeatures as set out in the attached claims but also any othercombination of features in the claims, wherein each feature mentioned inthe claims can be combined with any other feature or combination ofother features in the claims. Furthermore, any of the embodiments andfeatures described or depicted herein can be claimed in a separate claimand/or in any combination with any embodiment or feature described ordepicted herein or with any of the features of the attached claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a clustered virtualization environment according toparticular embodiments.

FIG. 1B illustrates data flow within a clustered virtualizationenvironment according to particular embodiments.

FIG. 2A illustrates a clustered virtualization environment implementinga virtualized file server according to particular embodiments.

FIG. 2B illustrates a clustered virtualization environment implementinga virtualized file server in which files used by user VMs are storedlocally on the same host machines as the user VMs according toparticular embodiments.

FIG. 3A illustrates an example hierarchical structure of a file serverinstance in a cluster according to particular embodiments.

FIG. 3B illustrates two example virtualized file server host machines,each providing file storage services for portions of two file serverinstances according to particular embodiments.

FIG. 3C illustrates example interactions between a client and hostmachines on which different portions of a virtualized file serverinstance are stored according to particular embodiments.

FIG. 3D illustrates an example virtualized file server providingfailover capability according to particular embodiments.

FIG. 3E illustrates an example virtualized file server that hasrecovered from a Controller/Service VM failure by switching to analternate Controller/Service VM according to particular embodiments.

FIG. 3F illustrates an example virtualized file server that hasrecovered from failure of a file server VM by electing a new leader fileserver VM according to particular embodiments.

FIG. 3G illustrates an example failure of a host machine that causesfailure of both the file server VM and the Controller/Service VM locatedon the host machine according to particular embodiments.

FIG. 311 illustrates an example virtualized file server that hasrecovered from a host machine failure by switching to aController/Service VM and a file server VM located on a backup hostmachine according to particular embodiments.

FIGS. 4A and 4B illustrate an example hierarchical namespace of a fileserver according to particular embodiments.

FIG. 4C illustrates distribution of stored data amongst host machines ina virtualized file server according to particular embodiments.

FIG. 5 illustrates an example method for accessing data in a virtualizedfile server according to particular embodiments.

FIG. 6 illustrates an example of how a file server may be deployedacross multiple clusters according to particular embodiments.

FIG. 7 illustrates a block diagram of a computing system suitable forimplementing particular embodiments.

FIG. 8 illustrates an example method for deploying a virtualized fileserver.

FIG. 9 illustrates an example method for upgrading a virtualized fileserver.

FIG. 10 illustrates an example method for performing a rolling upgradeof a virtualized file server.

FIG. 11 illustrates an example method for ingesting data into avirtualized file server.

FIG. 12 illustrates example virtualized file server operations.

FIG. 13 illustrates an example method for splitting an existingvirtualized file server into two or more new virtualized file servers.

FIG. 14 illustrates an example method for merging one or more existingvirtualized file servers to form a single virtualized file server.

FIG. 15 illustrates an example method for accessing shared filesystemson multiple virtualized file servers.

FIG. 16 illustrates an example method for recovering data after failureof a virtualized file server.

FIG. 17 illustrates an example interaction diagram for serving fileaccess requests in a virtualized file server geographically distributedacross clusters.

FIG. 18 illustrates an example method for detecting and recovering fromdata corruption in a virtualized file server.

FIG. 19 illustrates an example method for backing up data stored on avirtualized file server to cloud-based storage.

FIG. 20 illustrates an example method for storing virtualized fileserver data in tiers of cloud storage having different accesscharacteristics.

FIG. 21 illustrates an example method for recovering from block failuresin a virtualized file server.

FIG. 22 illustrates an example method for recovering from multi-nodefile service failures in a virtualized file server.

FIG. 23 illustrates an example method for providing storage systeminformation about a virtualized file server.

FIG. 24 illustrates an example method for providing high availability ofstorage services in a VFS.

DESCRIPTION OF EXAMPLE EMBODIMENTS

FIG. 1A illustrates a clustered virtualization environment 100 accordingto particular embodiments. The architecture of FIG. 1A can beimplemented for a distributed platform that contains multiple hostmachines 10 l a-c that manage multiple tiers of storage. The multipletiers of storage may include storage that is accessible through network140, such as, by way of example and not limitation, cloud storage 126(e.g., which may be accessible through the Internet), network-attachedstorage (NAS) 128 (e.g., which may be accessible through a LAN), or astorage area network (SAN). Unlike the prior art, the present embodimentalso permits local storage 122 a-c that is incorporated into or directlyattached to the host machine and/or appliance to be managed as part ofstorage pool 160. Examples of such local storage include Solid StateDrives 125 (henceforth “SSDs”), Hard Disk Drives 127 (henceforth “HDDs”or “spindle drives”), optical disk drives, external drives (e.g., astorage device connected to a host machine via a native drive interfaceor a serial attached SCSI interface), or any other direct-attachedstorage. These storage devices, both direct-attached andnetwork-accessible, collectively form storage pool 160. Virtual disks(or “vDisks”) may be structured from the physical storage devices instorage pool 160, as described in more detail below. As used herein, theterm vDisk refers to the storage abstraction that is exposed by aController/Service VM (CVM) 110 to be used by a user VM 105. Inparticular embodiments, the vDisk may be exposed via iSCSI (“internetsmall computer system interface”) or NFS (“network filesystem”) and ismounted as a virtual disk on the user VM. In particular embodiments,vDisks may be organized into one or more volume groups (VGs).

Each host machine 101 a-c may run virtualization software, such asVMWARE ESX(I), MICROSOFT HYPER-V, or REDHAT KVM. The virtualizationsoftware includes hypervisor 130 a-c to create, manage, and destroy userVMs 105, as well as managing the interactions between the underlyinghardware and user VMs 105. User VMs 105 may run one or more applicationsthat may operate as “clients” with respect to other elements withinvirtualization environment 100. Though not depicted in FIG. 1A, ahypervisor may connect to network 140. In particular embodiments, a hostmachine 101 may be a physical hardware computing device; in particularembodiments, a host machine 101 may be a virtual machine.

CVMs 110 a-c are used to manage storage and input/output (“I/O”)activities according to particular embodiments. These special VMs act asthe storage controller in the currently described architecture. Multiplesuch storage controllers may coordinate within a cluster to form aunified storage controller system. CVMs 110 may run as virtual machineson the various host machines 101, and work together to form adistributed system 110 that manages all the storage resources, includinglocal storage 122, NAS 128, and cloud storage 126. The CVMs may connectto network 140 directly, or via a hypervisor. Since the CVMs runindependent of hypervisors 130 a-c, this means that the current approachcan be used and implemented within any virtual machine architecture,since the CVMs of particular embodiments can be used in conjunction withany hypervisor from any virtualization vendor.

A host machine may be designated as a leader node within a cluster ofhost machines. For example, host machine 101 b, as indicated by theasterisks, may be a leader node. A leader node may have a softwarecomponent designated to perform operations of the leader. For example,CVM 110 b on host machine 101 b may be designated to perform suchoperations. A leader may be responsible for monitoring or handlingrequests from other host machines or software components on other hostmachines throughout the virtualized environment. If a leader fails, anew leader may be designated. In particular embodiments, a managementmodule (e.g., in the form of an agent) may be running on the leadernode.

Each CVM 110 a-c exports one or more block devices or NFS server targetsthat appear as disks to user VMs 105 a-c. These disks are virtual, sincethey are implemented by the software running inside CVMs 110 a-c. Thus,to user VMs 105 a-c, CVMs 110 a-c appear to be exporting a clusteredstorage appliance that contains some disks. All user data (including theoperating system) in the user VMs 105 a-c and reside on these virtualdisks.

Significant performance advantages can be gained by allowing thevirtualization system to access and utilize local storage 122 asdisclosed herein. This is because I/O performance is typically muchfaster when performing access to local storage 122 as compared toperforming access to NAS 128 across a network 140. This fasterperformance for locally attached storage 122 can be increased evenfurther by using certain types of optimized local storage devices, suchas SSDs. Further details regarding methods and mechanisms forimplementing the virtualization environment illustrated in FIG. 1A aredescribed in U.S. Pat. No. 8,601,473, which is hereby incorporated byreference in its entirety.

FIG. 1B illustrates data flow within an example clustered virtualizationenvironment 100 according to particular embodiments. As described above,one or more user VMs and a CVM may run on each host machine 101 alongwith a hypervisor. As a user VM performs I/O operations (e.g., a readoperation or a write operation), the I/O commands of the user VM may besent to the hypervisor that shares the same server as the user VM. Forexample, the hypervisor may present to the virtual machines an emulatedstorage controller, receive an I/O command and facilitate theperformance of the I/O command (e.g., via interfacing with storage thatis the object of the command, or passing the command to a service thatwill perform the I/O command). An emulated storage controller mayfacilitate I/O operations between a user VM and a vDisk. A vDisk maypresent to a user VM as one or more discrete storage drives, but eachvDisk may correspond to any part of one or more drives within storagepool 160. Additionally or alternatively, CVM 110 a-c may present anemulated storage controller either to the hypervisor or to user VMs tofacilitate I/O operations. CVM 110 a-c may be connected to storagewithin storage pool 160. CVM 110 a may have the ability to perform I/Ooperations using local storage 122 a within the same host machine 101 a,by connecting via network 140 to cloud storage 126 or NAS 128, or byconnecting via network 140 to local storage 122 b-c within another hostmachine 101 b-c (e.g., via connecting to another CVM 110 b or 110 c). Inparticular embodiments, any suitable computing system 700 may be used toimplement a host machine 101.

FIG. 2A illustrates a clustered virtualization environment 200implementing a virtualized file server (VFS) 202 according to particularembodiments. In particular embodiments, the VFS 202 provides fileservices to user VMs 105. The file services may include storing andretrieving data persistently, reliably, and efficiently. The uservirtual machines 105 may execute user processes, such as officeapplications or the like, on host machines 201 a-c. The stored data maybe represented as a set of storage items, such as files organized in ahierarchical structure of folders (also known as directories), which cancontain files and other folders, and shares, which can also containfiles and folders.

In particular embodiments, the VFS 202 may include a set of File ServerVirtual Machines (FSVMs) 170 a-c that execute on host machines 201 a-cand process storage item access operations requested by user VMs 201 a-cexecuting on the host machines 201 a-c. The FSVMs 170 a-c maycommunicate with storage controllers provided by CVMs 110 a-c executingon the host machines 201 a-c to store and retrieve files, folders, SMBshares, or other storage items on local storage 122 a-c associated with,e.g., local to, the host machines 201 a-c. The FSVMs 170 a-c may storeand retrieve block-level data on the host machines 201 a-c, e.g., on thelocal storage 122 a-c of the host machines 201 a-c. The block-level datamay include block-level representations of the storage items. Thenetwork protocol used for communication between user VMs 105, FSVMs 170a-c, and CVMs 110 a-c via the network 140 may be Internet Small ComputerSystems Interface (iSCSI), Server Message Block (SMB), NetworkFilesystem (NFS), pNFS (Parallel NFS), or another appropriate protocol.

For the purposes of VFS 202, host machine 201 c may be designated as aleader node within a cluster of host machines. In this case, FSVM 170 con host machine 101 c may be designated to perform such operations. Aleader may be responsible for monitoring or handling requests from FSVMson other host machines throughout the virtualized environment. If FSVM170 c fails, a new leader may be designated for VFS 202.

In particular embodiments, the user VMs 105 may send data to the VFS 202using write requests, and may receive data from it using read requests.The read and write requests, and their associated parameters, data, andresults, may be sent between a user VM 105 a and one or more file serverVMs (FSVMs) 170 a-c located on the same host machine as the user VM 105a or on different host machines from the user VM 105 a. The read andwrite requests may be sent between host machines 201 a-c via network140, e.g., using a network communication protocol such as iSCSI, CIFS,SMB, TCP, IP, or the like. When a read or write request is sent betweentwo VMs located on the same one of the host machines 201 a-c (e.g.,between the user VM 105 a and the FSVM 170 a located on the host machine201 a), the request may be sent using local communication within thehost machine 201 a instead of via the network 140. As described above,such local communication may be substantially faster than communicationvia the network 140. The local communication may be performed by, e.g.,writing to and reading from shared memory accessible by the user VM 105a and the FSVM 170 a, sending and receiving data via a local “loopback”network interface, local stream communication, or the like.

In particular embodiments, the storage items stored by the VFS 202, suchas files and folders, may be distributed amongst multiple FSVMs 170 a-c.In particular embodiments, when storage access requests are receivedfrom the user VMs 105, the VFS 202 identifies FSVMs 170 a-c at whichrequested storage items, e.g., folders, files, or portions thereof, arestored, and directs the user VMs 105 to the locations of the storageitems. The FSVMs 170 may maintain a storage map, such as a sharding map360 (shown in FIG. 3C), that maps names or identifiers of storage itemsto their corresponding locations. The storage map may be a distributeddata structure of which copies are maintained at each FSVM 170 a-c andaccessed using distributed locks or other storage item accessoperations. Alternatively, the storage map may be maintained by a leadernode such as the FSVM 170 c, and the other FSVMs 170 a and 170 b maysend requests to query and update the storage map to the leader FSVM 170c. Other implementations of the storage map are possible usingappropriate techniques to provide asynchronous data access to a sharedresource by multiple readers and writers. The storage map may map namesor identifiers of storage items in the form of text strings or numericidentifiers, such as folder names, files names, and/or identifiers ofportions of folders or files (e.g., numeric start offset positions andcounts in bytes or other units) to locations of the files, folders, orportions thereof. Locations may be represented as names of FSVMs 170a-c, e.g., “FSVM-1”, as network addresses of host machines 201 a-c onwhich FSVMs 170 a-c are located (e.g., “ip-addr1” or 128.1.1.10), or asother types of location identifiers.

When a user application executing in a user VM 105 a on one of the hostmachines 201 a initiates a storage access operation, such as reading orwriting data, the user VM 105 a may send the storage access operation ina request to one of the FSVMs 170 a-c on one of the host machines 201a-c. A FSVM 170 b executing on a host machine 201 b that receives astorage access request may use the storage map to determine whether therequested file or folder is located on the FSVM 170 b. If the requestedfile or folder is located on the FSVM 170 b, the FSVM 170 b executes therequested storage access operation. Otherwise, the FSVM 170 b respondsto the request with an indication that the data is not on the FSVM 170b, and may redirect the requesting user VM 105 a to the FSVM on whichthe storage map indicates the file or folder is located. The client maycache the address of the FSVM on which the file or folder is located, sothat it may send subsequent requests for the file or folder directly tothat FSVM.

As an example and not by way of limitation, the location of a file or afolder may be pinned to a particular FSVM 170 a by sending a fileservice operation that creates the file or folder to a CVM 110 aassociated with (e.g., located on the same host machine 201 a as) theFSVM 170 a. The CVM 110 a subsequently processes file service commandsfor that file for the FSVM 170 a and sends corresponding storage accessoperations to storage devices associated with the file. The CVM 110 amay associate local storage 122 a with the file if there is sufficientfree space on local storage 122 a. Alternatively, the CVM 110 a mayassociate a storage device located on another host machine 201 b, e.g.,in local storage 122 b, with the file under certain conditions, e.g., ifthere is insufficient free space on the local storage 122 a, or ifstorage access operations between the CVM 110 a and the file areexpected to be infrequent. Files and folders, or portions thereof, mayalso be stored on other storage devices, such as the network-attachedstorage (NAS) 128 or the cloud storage 126 of the storage pool 160.

In particular embodiments, a name service 220, such as that specified bythe Domain Name System (DNS) Internet protocol, may communicate with thehost machines 201 a-c via the network 140 and may store a database ofdomain name (e.g., host name) to IP address mappings. The domain namesmay correspond to FSVMs 170 a-c, e.g., fsvm1.domain.com orip-addr1.domain.com for an FSVM named FSVM-1. The name service 220 maybe queried by the user VMs 105 to determine the IP address of aparticular host machine 201 a-c given a name of the host machine, e.g.,to determine the IP address of the host name ip-addr1 for the hostmachine 201 a. The name service 220 may be located on a separate servercomputer system or on one or more of the host machines 201. The namesand IP addresses of the host machines of the VFS instance 202, e.g., thehost machines 201, may be stored in the name service 220 so that theuser VMs 105 may determine the IP address of each of the host machines201 or FSVMs 170. The name of each VFS instance 202, e.g., FS1, FS2, orthe like, may be stored in the name service 220 in association with aset of one or more names that contains the name(s) of the host machines201 or FSVMs 170 of the VFS instance 202. The FSVMs 170 a-c may beassociated with the host names ip-addr1, ip-addr2, and ip-addr3,respectively. For example, the file server instance name FS1.domain.commay be associated with the host names ip-addr1, ip-addr2, and ip-addr3in the name service 220, so that a query of the name service 220 for theserver instance name “FS1” or “FS1.domain.com” returns the namesip-addr1, ip-addr2, and ip-addr3. As another example, the file serverinstance name FS1.domain.com may be associated with the host namesfsvm-1, fsvm-2, and fsvm-3. Further, the name service 220 may return thenames in a different order for each name lookup request, e.g., usinground-robin ordering, so that the sequence of names (or addresses)returned by the name service for a file server instance name is adifferent permutation for each query until all the permutations havebeen returned in response to requests, at which point the permutationcycle starts again, e.g., with the first permutation. In this way,storage access requests from user VMs 105 may be balanced across thehost machines 201, since the user VMs 105 submit requests to the nameservice 220 for the address of the VFS instance 202 for storage itemsfor which the user VMs 105 do not have a record or cache entry, asdescribed below.

In particular embodiments, each FSVM 170 may have two IP addresses: anexternal IP address and an internal IP address. The external IPaddresses may be used by SMB/CIFS clients, such as user VMs 105, toconnect to the FSVMs 170. The external IP addresses may be stored in thename service 220. The IP addresses ip-addr1, ip-addr2, and ip-addr3described above are examples of external IP addresses. The internal IPaddresses may be used for iSCSI communication to CVMs 110, e.g., betweenthe FSVMs 170 and the CVMs 110. Other internal communications may besent via the internal IP addresses as well, e.g., file serverconfiguration information may be sent from the CVMs 110 to the FSVMs 170using the internal IP addresses, and the CVMs 110 may get file serverstatistics from the FSVMs 170 via internal communication as needed.

Since the VFS 202 is provided by a distributed set of FSVMs 170 a-c, theuser VMs 105 that access particular requested storage items, such asfiles or folders, do not necessarily know the locations of the requestedstorage items when the request is received. A distributed file systemprotocol, e.g., MICROSOFT DFS or the like, is therefore used, in which auser VM 105 a may request the addresses of FSVMs 170 a-c from a nameservice 220 (e.g., DNS). The name service may send one or more networkaddresses of FSVMs 170 a-c to the user VM 105 a, in an order thatchanges for each subsequent request. These network addresses are notnecessarily the addresses of the FSVM 170 b on which the storage itemrequested by the user VM 105 a is located, since the name service 220does not necessarily have information about the mapping between storageitems and FSVMs 170 a-c. Next, the user VM 170 a may send an accessrequest to one of the network addresses provided by the name service,e.g., the address of FSVM 170 b. The FSVM 170 b may receive the accessrequest and determine whether the storage item identified by the requestis located on the FSVM 170 b. If so, the FSVM 170 b may process therequest and send the results to the requesting user VM 105 a. However,if the identified storage item is located on a different FSVM 170 c,then the FSVM 170 b may redirect the user VM 105 a to the FSVM 170 c onwhich the requested storage item is located by sending a “redirect”response referencing FSVM 170 c to the user VM 105 a. The user VM 105 amay then send the access request to FSVM 170 c, which may perform therequested operation for the identified storage item.

A particular VFS 202, including the items it stores, e.g., files andfolders, may be referred to herein as a VFS “instance” 202 and may havean associated name, e.g., FS1, as described above. Although a VFSinstance 202 may have multiple FSVMs 170 distributed across differenthost machines 201, with different files being stored on FSVMs 170, theVFS instance 202 may present a single name space to its clients such asthe user VMs 105. The single name space may include, for example, a setof named “shares” and each share may have an associated folder hierarchyin which files are stored. Storage items such as files and folders mayhave associated names and metadata such as permissions, access controlinformation, size quota limits, file types, files sizes, and so on. Asanother example, the name space may be a single folder hierarchy, e.g.,a single root directory that contains files and other folders. User VMs105 may access the data stored on a distributed VFS instance 202 viastorage access operations, such as operations to list folders and filesin a specified folder, create a new file or folder, open an existingfile for reading or writing, and read data from or write data to a file,as well as storage item manipulation operations to rename, delete, copy,or get details, such as metadata, of files or folders. Note that foldersmay also be referred to herein as “directories.”

In particular embodiments, storage items such as files and folders in afile server namespace may be accessed by clients such as user VMs 105 byname, e.g., “\Folder-1 \File-1” and “\Folder-2\File-2” for two differentfiles named File-1 and File-2 in the folders Folder-1 and Folder-2,respectively (where Folder-1 and Folder-2 are sub-folders of the rootfolder). Names that identify files in the namespace using folder namesand file names may be referred to as “path names.” Client systems mayaccess the storage items stored on the VFS instance 202 by specifyingthe file names or path names, e.g., the path name “\Folder-1 \File-1”,in storage access operations. If the storage items are stored on a share(e.g., a shared drive), then the share name may be used to access thestorage items, e.g., via the path name “\\Share-1 \Folder-1 \File-1” toaccess File-1 in folder Folder-1 on a share named Share-1.

In particular embodiments, although the VFS instance 202 may storedifferent folders, files, or portions thereof at different locations,e.g., on different FSVMs 170, the use of different FSVMs 170 or otherelements of storage pool 160 to store the folders and files may behidden from the accessing clients. The share name is not necessarily aname of a location such as an FSVM 170 or host machine 201. For example,the name Share-1 does not identify a particular FSVM 170 on whichstorage items of the share are located. The share Share-1 may haveportions of storage items stored on three host machines 201 a-c, but auser may simply access Share-1, e.g., by mapping Share-1 to a clientcomputer, to gain access to the storage items on Share-1 as if they werelocated on the client computer. Names of storage items, such as filenames and folder names, are similarly location-independent. Thus,although storage items, such as files and their containing folders andshares, may be stored at different locations, such as different hostmachines 201 a-c, the files may be accessed in a location-transparentmanner by clients (such as the user VMs 105). Thus, users at clientsystems need not specify or know the locations of each storage itembeing accessed. The VFS 202 may automatically map the file names, foldernames, or full path names to the locations at which the storage itemsare stored. As an example and not by way of limitation, a storage item'slocation may be specified by the name, address, or identity of the FSVM170 a-c that provides access to the storage item on the host machine 201a-c on which the storage item is located. A storage item such as a filemay be divided into multiple parts that may be located on differentFSVMs 170, in which case access requests for a particular portion of thefile may be automatically mapped to the location of the portion of thefile based on the portion of the file being accessed (e.g., the offsetfrom the beginning of the file and the number of bytes being accessed).

In particular embodiments, VFS 202 determines the location, e.g., FSVM170, at which to store a storage item when the storage item is created.For example, a FSVM 170 a may attempt to create a file or folder using aController/Service VM 110 a on the same host machine 201 a as the userVM 105 a that requested creation of the file, so that theController/Service VM 110 a that controls access operations to the filefolder is co-located with the user VM 105 a. In this way, since the userVM 105 a is known to be associated with the file or folder and is thuslikely to access the file again, e.g., in the near future or on behalfof the same user, access operations may use local communication orshort-distance communication to improve performance, e.g., by reducingaccess times or increasing access throughput. If there is a local CVM110 a on the same host machine as the FSVM 170 a, the FSVM 170 a mayidentify it and use it by default. If there is no local CVM 110 a on thesame host machine as the FSVM 170 a, a delay may be incurred forcommunication between the FSVM 170 a and a CVM 110 b on a different hostmachine 201 b. Further, the VFS 202 may also attempt to store the fileon a storage device that is local to the CVM 110 a being used to createthe file, such as local storage 122 a, so that storage access operationsbetween the CVM 110 a and local storage 122 a may use local orshort-distance communication.

In particular embodiments, if a CVM 110 a is unable to store the storageitem in local storage 122 a of a host machine 201 a on which an FSVM 170a, e.g., because local storage 122 a does not have sufficient availablefree space, then the file may be stored in local storage 122 b of adifferent host machine 201 b. In this case, the stored file is notphysically local to the host machine 201 a, but storage accessoperations for the file are performed by the locally-associated CVM 110a and FSVM 170 a, and the CVM 110 a may communicate with local storage122 b on the remote host machine 201 b using a network file sharingprotocol, e.g., iSCSI, SAMBA, or the like.

In particular embodiments, if a virtual machine, such as a user VM 105a, CVM 110 a, or FSVM 170 a, moves from a host machine 201 a to adestination host machine 201 b, e.g., because of resource availabilitychanges, and data items such as files or folders associated with the VMare not locally accessible on the destination host machine 201 b, thendata migration may be performed for the data items associated with themoved VM to migrate them to the new host machine 201 b, so that they arelocal to the moved VM on the new host machine 201 b. FSVMs 170 maydetect removal and addition of CVMs 110 (as may occur, for example, whena CVM 110 fails or is shut down) via the iSCSI protocol or othertechnique, such as heartbeat messages. As another example, a FSVM 170may determine that a particular file's location is to be changed, e.g.,because a disk on which the file is stored is becoming full, becausechanging the file's location is likely to reduce network communicationdelays and therefore improve performance, or for other reasons. Upondetermining that a file is to be moved, VFS 202 may change the locationof the file by, for example, copying the file from its existinglocation(s), such as local storage 122 a of a host machine 201 a, to itsnew location(s), such as local storage 122 b of host machine 201 b (andto or from other host machines, such as local storage 122 c of hostmachine 201 c if appropriate), and deleting the file from its existinglocation(s). Write operations on the file may be blocked or queued whilethe file is being copied, so that the copy is consistent. The VFS 202may also redirect storage access requests for the file from an FSVM 170a at the file's existing location to a FSVM 170 b at the file's newlocation.

In particular embodiments, VFS 202 includes at least three File ServerVirtual Machines (FSVMs) 170 a-c located on three respective hostmachines 201 a-c. To provide high-availability, there may be a maximumof one FSVM 170 a for a particular VFS instance 202 per host machine 201in a cluster. If two FSVMs 170 are detected on a single host machine201, then one of the FSVMs 170 may be moved to another host machineautomatically, or the user (e.g., system administrator) may be notifiedto move the FSVM 170 to another host machine. The user may move a FSVM170 to another host machine using an administrative interface thatprovides commands for starting, stopping, and moving FSVMs 170 betweenhost machines 201.

In particular embodiments, two FSVMs 170 of different VFS instances 202may reside on the same host machine 201 a. If the host machine 201 afails, the FSVMs 170 on the host machine 201 a become unavailable, atleast until the host machine 201 a recovers. Thus, if there is at mostone FSVM 170 for each VFS instance 202 on each host machine 201 a, thenat most one of the FSVMs 170 may be lost per VFS 202 per failed hostmachine 201. As an example, if more than one FSVM 170 for a particularVFS instance 202 were to reside on a host machine 201 a, and the VFSinstance 202 includes three host machines 201 a-c and three FSVMs 170,then loss of one host machine would result in loss of two-thirds of theFSVMs 170 for the VFS instance 202, which would be more disruptive andmore difficult to recover from than loss of one-third of the FSVMs 170for the VFS instance 202.

In particular embodiments, users, such as system administrators or otherusers of the user VMs 105, may expand the cluster of FSVMs 170 by addingadditional FSVMs 170. Each FSVM 170 a may be associated with at leastone network address, such as an IP (Internet Protocol) address of thehost machine 201 a on which the FSVM 170 a resides. There may bemultiple clusters, and all FSVMs of a particular VFS instance areordinarily in the same cluster. The VFS instance 202 may be a member ofa MICROSOFT ACTIVE DIRECTORY domain, which may provide authenticationand other services such as name service 220.

FIG. 2B illustrates data flow within a clustered virtualizationenvironment 200 implementing a VFS instance 202 in which stored itemssuch as files and folders used by user VMs 105 are stored locally on thesame host machines 201 as the user VMs 105 according to particularembodiments. As described above, one or more user VMs 105 and aController/Service VM 110 may run on each host machine 201 along with ahypervisor 130. As a user VM 105 processes I/O commands (e.g., a read orwrite operation), the I/O commands may be sent to the hypervisor 130 onthe same server or host machine 201 as the user VM 105. For example, thehypervisor 130 may present to the user VMs 105 a VFS instance 202,receive an I/O command, and facilitate the performance of the I/Ocommand by passing the command to a FSVM 170 that performs the operationspecified by the command. The VFS 202 may facilitate I/O operationsbetween a user VM 105 and a virtualized filesystem. The virtualizedfilesystem may appear to the user VM 105 as a namespace of mappableshared drives or mountable network filesystems of files and directories.The namespace of the virtualized filesystem may be implemented usingstorage devices in the local storage 122, such as disks 204, onto whichthe shared drives or network filesystems, files, and folders, orportions thereof, may be distributed as determined by the FSVMs 170. TheVFS 202 may thus provide features disclosed herein, such as efficientuse of the disks 204, high availability, scalability, and others. Theimplementation of these features may be transparent to the user VMs 105.The FSVMs 170 may present the storage capacity of the disks 204 of thehost machines 201 as an efficient, highly-available, and scalablenamespace in which the user VMs 105 may create and access shares, files,folders, and the like.

As an example, a network share may be presented to a user VM 105 as oneor more discrete virtual disks, but each virtual disk may correspond toany part of one or more virtual or physical disks 204 within storagepool 160. Additionally or alternatively, the FSVMs 170 may present a VFS202 either to the hypervisor 130 or to user VMs 105 of a host machine201 to facilitate I/O operations. The FSVMs 170 may access the localstorage 122 via Controller/Service VMs 110. As described above withreference to FIG. 1B, a Controller/Service VM 110 a may have the abilityto perform I/O operations using local storage 122 a within the same hostmachine 201 a by connecting via the network 140 to cloud storage 126 orNAS 128, or by connecting via the network 140 to local storage 122 b-cwithin another host machine 201 b-c (e.g., by connecting to anotherController/Service VM 110 b-c).

In particular embodiments, each user VM 105 may access one or morevirtual disk images 206 stored on one or more disks 204 of the localstorage 122, the cloud storage 126, and/or the NAS 128. The virtual diskimages 206 may contain data used by the user VMs 105, such as operatingsystem images, application software, and user data, e.g., user homefolders and user profile folders. For example, FIG. 2B illustrates threevirtual machine images 206 a-c. The virtual machine image 206 a may be afile named UserVM105 a.vmdisk (or the like) stored on disk 204 a oflocal storage 122 a of host machine 201 a. The virtual machine image 206a may store the contents of the user VM 105 a's hard drive. The disk 204a on which the virtual machine image 206 a is “local to” the user VM 105a on host machine 201 a because the disk 204 a is in local storage 122 aof the host machine 201 a on which the user VM 105 a is located. Thus,the user VM 105 a may use local (intra-host machine) communication toaccess the virtual machine image 206 a more efficiently, e.g., with lesslatency and higher throughput, than would be the case if the virtualmachine image 206 a were stored on disk 204 b of local storage 122 b ofa different host machine 201 b, because inter-host machine communicationacross the network 140 would be used in the latter case. Localcommunication within a host machine 201 a is described in further detailwith reference to FIG. 4C. Similarly, a virtual machine image 206 b,which may be a file named UserVM105 b.vmdisk (or the like), is stored ondisk 204 b of local storage 122 b of host machine 201 b, and the image206 b is local to the user VM 105 b located on host machine 201 b. Thus,the user VM 105 a may access the virtual machine image 206 b moreefficiently than the virtual machine 206 a on host machine 201 a, forexample. In another example, the CVM 110 c may be located on the samehost machine 201 c as the user VM 105 c that accesses a virtual machineimage 206 c (UserVM105 c.vmdisk) of the user VM 105 c, with the virtualmachine image file 206 c being stored on a different host machine 201 bthan the user VM 105 c and the CVM 110 c. In this example, communicationbetween the user VM 105 c and the CVM 110 c may still be local, e.g.,more efficient than communication between the user VM 105 c and a CVM110 b on a different host machine 201 b, but communication between theCVM 110 c and the disk 204 b on which the virtual machine image 206 c isstored is via the network 140, as shown by the dashed lines between CVM110 c and the network 140 and between the network 140 and local storage122 b. The communication between CVM 110 c and the disk 204 b is notlocal, and thus may be less efficient than local communication such asmay occur between the CVM 110 c and a disk 204 c in local storage 122 cof host machine 201 c. Further, a user VM 105 c on host machine 201 cmay access data such as the virtual disk image 206 c stored on a remote(e.g., non-local) disk 204 b via network communication with a CVM 110 blocated on the remote host machine 201 b. This case may occur if CVM 110c is not present on host machine 201 c, e.g., because CVM 110 c hasfailed, or if the FSVM 170 c has been configured to communicate withlocal storage 122 b on host machine 201 b via the CVM 110 b on hostmachine 201 b, e.g., to reduce computational load on host machine 201 c.

In particular embodiments, since local communication is expected to bemore efficient than remote communication, the FSVMs 170 may storestorage items, such as files or folders, e.g., the virtual disk images206, as block-level data on local storage 122 of the host machine 201 onwhich the user VM 105 that is expected to access the files is located. Auser VM 105 may be expected to access particular storage items if, forexample, the storage items are associated with the user VM 105, such asby configuration information. For example, the virtual disk image 206 amay be associated with the user VM 105 a by configuration information ofthe user VM 105 a. Storage items may also be associated with a user VM105 via the identity of a user of the user VM 105. For example, filesand folders owned by the same user ID as the user who is logged into theuser VM 105 a may be associated with the user VM 105 a. If the storageitems expected to be accessed by a user VM 105 a are not stored on thesame host machine 201 a as the user VM 105 a, e.g., because ofinsufficient available storage capacity in local storage 122 a of thehost machine 201 a, or because the storage items are expected to beaccessed to a greater degree (e.g., more frequently or by more users) bya user VM 105 b on a different host machine 201 b, then the user VM 105a may still communicate with a local CVM 110 a to access the storageitems located on the remote host machine 201 b, and the local CVM 110 amay communicate with local storage 122 b on the remote host machine 201b to access the storage items located on the remote host machine 201 b.If the user VM 105 a on a host machine 201 a does not or cannot use alocal CVM 110 a to access the storage items located on the remote hostmachine 201 b, e.g., because the local CVM 110 a has crashed or the userVM 105 a has been configured to use a remote CVM 110 b, thencommunication between the user VM 105 a and local storage 122 b on whichthe storage items are stored may be via a remote CVM 110 b using thenetwork 140, and the remote CVM 110 b may access local storage 122 busing local communication on host machine 201 b. As another example, auser VM 105 a on a host machine 201 a may access storage items locatedon a disk 204 c of local storage 122 c on another host machine 201 c viaa CVM 110 b on an intermediary host machine 201 b using networkcommunication between the host machines 201 a and 201 b and between thehost machines 201 b and 201 c.

FIG. 3A illustrates an example hierarchical structure 300 of a VFSinstance in a cluster according to particular embodiments. A Cluster 302contains two FSVMs, FSVM1 304 and FSVM2 306. Each FSVM may be identifiedby a name such as “\\instance”, e.g., “\\FS1” for WINDOWS filesystems,or a name such as “instance”, e.g., “FS1” for UNIX-type filesystems.FSVM1 304 contains shares, including Share-1 308 and Share-2 310. Sharesmay have names such as “Users” for a share that stores user homedirectories, or the like. Each share may have a path name such as \\Share-1 or \\ Users. As an example and not by way of limitation, a sharemay correspond to a disk partition or a pool of filesystem blocks onWINDOWS and UNIX-type filesystems. As another example and not by way oflimitation, a share may correspond to a folder or directory on a FSVM.Shares may appear in the filesystem instance 202 as folders ordirectories to users of user VMs 105. Share-1 308 includes two folders,Folder-1 312, and Folder-2 314, and may also include one or more files(e.g., files not in folders). Each folder 312, 314 may include one ormore files 318. Share-2 310 includes a folder Folder-3 316, whichincludes a file File-2 320. Each folder has a folder name such as“Folder-1”, “Users”, or “Sam” and a path name such as “\\Share-1\Folder-1” (WINDOWS) or “share-1:/Users/Sam” (UNIX). Similarly, eachfile has a file name such as “File-1” or “Forecast.xls” and a path namesuch as “\\ Share-1 \Folder-1 \File-1” or“share-1:/Users/Sam/Forecast.xls”.

FIG. 3B illustrates two example host machines 201 a and 201 b, eachproviding file storage services for portions of two VFS instances FS1and FS2 according to particular embodiments. The first host machine,Host-1 201 a, includes two user VMs 105 a, a Hypervisor 130 a, a FSVMnamed FileServer-VM-1 (abbreviated FSVM-1) 170 a, a Controller/ServiceVM named CVM-1 110 a, and local storage 122 a. Host-1 's FileServer-VM-1170 a has an IP (Internet Protocol) network address of 10.1.1.1, whichis an address of a network interface on Host-1 201 a. Host-1 has ahostname ip-addr1, which may correspond to Host-1 's IP address10.1.1.1. The second host machine, Host-2 201 b, includes two user VMs105 b, a Hypervisor 130 b, a File Server VM named FileServer-VM-2(abbreviated FSVM-2) 170 b, a Controller/Service VM named CVM-2 110 b,and local storage 122 b. Host-2 's FileServer-VM-1 170 b has an IPnetwork address of 10.1.1.2, which is an address of a network interfaceon Host-2 201 b.

In particular embodiments, filesystems FileSystem-1A 364 a andFileSystem-2A 365 a implement the structure of files and folders forportions of the FS1 and FS2 file server instances, respectively, thatare located on (e.g., served by) FileServer-VM-1 170 a on Host-1 201 a.Other filesystems on other host machines may implement other portions ofthe FS1 and FS2 file server instances. The filesystems 364 a and 365 amay implement the structure of at least a portion of a file serverinstance by translating filesystem operations, such as opening a file,writing data to or reading data from the file, deleting a file, and soon, to disk I/O operations such as seeking to a portion of the disk,reading or writing an index of file information, writing data to orreading data from blocks of the disk, allocating or de-allocating theblocks, and so on. The filesystems 364 a, 365 a may thus store theirfilesystem data, including the structure of the folder and filehierarchy, the names of the storage items (e.g., folders and files), andthe contents of the storage items on one or more storage devices, suchas local storage 122 a. The particular storage device or devices onwhich the filesystem data for each filesystem are stored may bespecified by an associated filesystem pool (e.g., 366 a-c and 367 a-c).For example, the storage device(s) on which data for FileSystem-1A 364 aand FileSystem-2A, 365 a are stored may be specified by respectivefilesystem pools FS1-Pool-1 366 a and FS2-Pool-2 367 a. The storagedevices for the pool 366 a may be selected from volume groups providedby CVM-1 110 a, such as volume group VG1 368 a and volume group VG2 369a. Each volume group 368 a, 369 a may include a group of one or moreavailable storage devices that are present in local storage 122 aassociated with (e.g., by iSCSI communication) the CVM-1 110 a. TheCVM-1 110 a may be associated with a local storage 122 a on the samehost machine 201 a as the CVM-1 110 a, or with a local storage 122 b ona different host machine 201 b. The CVM-1 110 a may also be associatedwith other types of storage, such as cloud storage 126, NAS 128 or thelike. Although the examples described herein include particular hostmachines, virtual machines, file servers, file server instances, fileserver pools, CVMs, volume groups, and associations therebetween, anynumber of host machines, virtual machines, file servers, file serverinstances, file server pools, CVMs, volume groups, and any associationstherebetween are possible and contemplated.

In particular embodiments, the filesystem pool 366 a may associate anystorage device in one of the volume groups 368 a, 369 a of storagedevices that are available in local storage 122 a with the filesystemFileSystem-1A 364 a. For example, the filesystem pool FS1-Pool-1 366 amay specify that a disk device named hd1 in the volume group VG1 368 aof local storage 122 a is a storage device for FileSystem-1A 364 a forfile server FS1 on FSVM-1 170 a. A filesystem pool FS2-Pool-2 367 a mayspecify a storage device FileSystem-2A 365 a for file server FS2 onFSVM-1 170 a. The storage device for FileSystem-2A 365 a may be, e.g.,the disk device hd1, or a different device in one of the volume groups368 a, 369 a, such as a disk device named hd2 in volume group VG2 369 a.Each of the filesystems FileSystem-1A 364 a, FileSystem-2A 365 a may be,e.g., an instance of the NTFS filesystem used by the WINDOWS operatingsystem, of the UFS Unix filesystem, or the like. The term “filesystem”may also be used herein to refer to an instance of a type of filesystem,e.g., a particular structure of folders and files with particular namesand content.

In one example, referring to FIG. 3A, an FS1 hierarchy rooted at FileServer FS1 304 may be located on FileServer-VM-1 170 a and stored infilesystem instance FileSystem-1A 364 a. That is, the filesysteminstance FileSystem-1A 364 a may store the names of the shares andstorage items (such as folders and files), as well as the contents ofthe storage items, shown in the hierarchy at and below File Server FS1304. A portion of the FS1 hierarchy shown in FIG. 3A, such the portionrooted at Folder-2 314, may be located on FileServer-VM-2-170 b onHost-2 201 b instead of FileServer-VM-1-170 a, in which case thefilesystem instance FileSystem-1B 364 b may store the portion of the FS1hierarchy rooted at Folder-2 314, including Folder-3 314, Folder-4 322and File-3 324. Similarly, an FS2 hierarchy rooted at File Server FS2306 in FIG. 3A may be located on FileServer-VM-1 170 a and stored infilesystem instance FileSystem-2A 365 a. The FS2 hierarchy may be splitinto multiple portions (not shown), such that one portion is located onFileServer-VM-1 170 a on Host-1 201 a, and another portion is located onFileServer-VM-2 170 b on Host-2 201 b and stored in filesystem instanceFileSystem-2B 365 b.

In particular embodiments, FileServer-VM-1 (abbreviated FSVM-1) 170 a onHost-1 201 a is a leader for a portion of file server instance FS1 and aportion of FS2, and is a backup for another portion of FS1 and anotherportion of FS2. The portion of FS1 for which FileServer-VM-1 170 a is aleader corresponds to a storage pool labeled FS1-Pool-1 366 a.FileServer-VM-1 is also a leader for FS2-Pool-2 367 a, and is a backup(e.g., is prepared to become a leader upon request, such as in responseto a failure of another FSVM) for FS1-Pool-3 366 b and FS2-Pool-4 367 bon Host-2. In particular embodiments, FileServer-VM-2 (abbreviatedFSVM-2) 170 b is a leader for a portion of file server instance FS1 anda portion of FS2, and is a backup for another portion of FS1 and anotherportion of FS2. The portion of FS1 for which FSVM-2 170 b is a leadercorresponds to a storage pool labeled FS1-Pool-3 366 b. FSVM-2 170 b isalso a leader for FS2-Pool-4 367 b, and is a backup for FS1-Pool-1 366 aand FS2-Pool-2 367 a on Host-1.

In particular embodiments, the file server instances FS1, FS2 providedby the FSVMs 170 a and 170 b may be accessed by user VMs 105 via anetwork filesystem protocol such as SMB, CIFS, NFS, or the like. EachFSVM 170 a and 170 b may provide what appears to client applications onuser VMs 105 to be a single filesystem instance, e.g., a singlenamespace of shares, files and folders, for each file server instance202. However, the shares, files, and folders in a file server instancesuch as FS1 may actually be distributed across multiple FSVMs 170 a and170 b. For example, different folders in the same file server instancemay be associated with different corresponding FSVMs 170 a and 170 b andCVMs 110 a and 110 b on different host machines 201 a and 201 b.

The example file server instance FS1 304 shown in FIG. 3A has twoshares, Share-1 308 and Share-2 310. Share-1 308 may be located onFSVM-1 170 a, CVM-1 110 a, and local storage 122 a. Network filesystemprotocol requests from user VMs 105 to read or write data on file serverinstance FS1 304 and any share, folder, or file in the instance may besent to FSVM-1 170 a. FSVM-1 170 a may determine whether the requesteddata, e.g., the share, folder, file, or a portion thereof, referenced inthe request, is located on FSVM-1, and FSVM-1 is a leader for therequested data. If not, FSVM-1 may respond to the requesting User-VMwith an indication that the requested data is not covered by (e.g., isnot located on or served by) FSVM-1. Otherwise, the requested data iscovered by (e.g., is located on or served by) FSVM-1, so FSVM-1 may sendiSCSI protocol requests to a CVM that is associated with the requesteddata. Note that the CVM associated with the requested data may be theCVM-1 110 a on the same host machine 201 a as the FSVM-1, or a differentCVM on a different host machine 201 b, depending on the configuration ofthe VFS 202. In this example, the requested Share-1 is located onFSVM-1, so FSVM-1 processes the request. To provide for pathavailability, multipath I/O (MPIO) may be used for communication withthe FSVM, e.g., for communication between FSVM-1 and CVM-1. The activepath may be set to the CVM that is local to the FSVM (e.g., on the samehost machine) by default. The active path may be set to a remote CVMinstead of the local CVM, e.g., when a failover occurs.

Continuing with the data request example, the associated CVM is CVM 110a, which may in turn access the storage device associated with therequested data as specified in the request, e.g., to write specifieddata to the storage device or read requested data from a specifiedlocation on the storage device. In this example, the associated storagedevice is in local storage 122 a, and may be an HDD or SSD. CVM-1 110 amay access the HDD or SSD via an appropriate protocol, e.g., iSCSI,SCSI, SATA, or the like. CVM 110 a may send the results of accessinglocal storage 122 a, e.g., data that has been read, or the status of adata write operation, to CVM 110 a via, e.g., SATA, which may in turnsend the results to FSVM-1 170 a via, e.g., iSCSI. FSVM-1 170 a may thensend the results to user VM 105 a via SMB through the Hypervisor 130 a.

Share-2 310 may be located on FSVM-2 170 b, on Host-2. Network fileservice protocol requests from user VMs 105 to read or write data onShare-2 may be directed to FSVM-2 170 b on Host-2 by other FSVMs 170 a.Alternatively, user VMs 105 may send such requests directly to FSVM-2170 b on Host-2, which may process the requests using CVM-2 110 b andlocal storage 122 b on Host-2 as described above for FSVM-1 170 a onHost-1.

A file server instance 202 such as FS1 304 in FIG. 3A may appear as asingle filesystem instance (e.g., a single namespace of folders andfiles that are accessible by their names or pathnames without regard fortheir physical locations), even though portions of the filesystem arestored on different host machines 201 a-c. Since each FSVM 170 mayprovide a portion of a file server instance 202, each FSVM 170 may haveone or more “local” filesystems 364 a, 365 a that provide the portion ofthe file server instance 202 (e.g., the portion of the namespace offiles and folders) associated with the FSVM 170.

FIG. 3C illustrates example interactions between a client 330 and FSVMs170 a and 170 c on which different portions of a VFS instance are storedaccording to particular embodiments. A client 330, e.g., an applicationprogram executing in one of the user VMs 105 on the host machines 201a-c of FIGS. 2A-2B (e.g., user VM 105 b on host machine 201 b) requestsaccess to a folder \\FS1.domain.name\ Share-1 \Folder-3. The request maybe in response to an attempt to map \\FS1.domain.name\ Share-1 to anetwork drive in the operating system executing in the user VM 105 cfollowed by an attempt to access the contents of Share-1 or to accessthe contents of Folder-3, such as listing the files in Folder-3.

FIG. 3C shows interactions that occur between the client 330, FSVMs 170a and 170 b on host machines 201 a and 201 b, and a name server 332 whena storage item is mapped or otherwise accessed. The name server 332 maybe provided by a server computer system, such as one or more of the hostmachines 201, or a server computer system separate from the hostmachines 201. In one example, the name server 332 may provide a nameservice or directory service (e.g., MICROSOFT ACTIVE DIRECTORY)executing on one or more computer systems and accessible via the network140. The name service may define a namespace for the network in order toassign names to storage items and network resources. By mapping thenames to their respective network addresses, particular embodiments maylocate, manage, administer, and organize storage items and networkresources (e.g., vdisks, volume groups, folders, files, printers, users,groups, devices, and other objects). When using the name service, a usermay simply provide a name in order to locate and access the storage itemor network resource (without having to specify a physical or virtualaddress). The interactions are shown as arrows that representcommunications, e.g., messages sent via the network 140. Note that theclient 330 may be executing in a user VM 105, which may be co-locatedwith one of the FSVMs 170 a and 170 b. In such a co-located case, thearrows between the client 330 and the host machine 201 on which the FSVM170 is located may represent communication within the host machine 201,and such intra-host machine communication may be performed using amechanism different from communication over the network 140, e.g.,shared memory or inter process communication.

In particular embodiments, when the client 330 requests access toFolder-3, a VFS client component executing in the user VM 105 b may usea distributed filesystem protocol such as MICROSOFT DFS, or the like, tosend the storage access request to one or more of the FSVMs 170 a-c ofFIGS. 2A-2B. To access the requested file or folder, the clientdetermines the location of the requested file or folder, e.g., theidentity and/or network address of the FSVM 170 on which the file orfolder is located. The client may query a domain cache of FSVM 170 a-cnetwork addresses that the client has previously identified (e.g.,looked up). If the domain cache contains the network address of an FSVM170 associated with the requested folder name \\FS1.domain.name\ Share-1\Folder-3, then the client retrieves the associated network address fromthe domain cache and sends the access request to the network address,starting at step 393 as described below.

In particular embodiments, at step 381, the client may send a requestfor a list of addresses of FSVMs 170 a-170 c to a name server 332. Thename server 332 may be, e.g., a DNS server or other type of server, suchas a MICROSOFT domain controller (not shown), that has a database ofFSVM addresses. At step 382, the name server 332 may send a reply thatcontains a list of FSVM 170 network addresses, e.g., ip-addr1, ip-addr2,and ip-addr3, which correspond to the FSVMs 170 a-c in this example. Atstep 383, the client 330 may send an access request to one of thenetwork addresses, e.g., the first network address in the list (ip-addr1in this example), requesting the contents of Folder-3 of Share-1. Byselecting the first network address in the list, the particular FSVM 170to which the access request is sent may be varied, e.g., in around-robin manner by enabling round-robin DNS (or the like) on the nameserver 332. The access request may be, e.g., an SMB connect request, anNFS open request, and/or appropriate request(s) to traverse thehierarchy of Share-1 to reach the desired folder or file, e.g., Folder-3in this example.

At step 384, FileServer-VM-1 170 a may process the request received atstep 383 by searching a mapping or lookup table, such as a sharding map360 a, for the desired folder or file. The map 360 maps storage items,such as shares, folders, or files, to their corresponding locations,e.g., the names or addresses of FSVMs 170. The map 360 may have the samecontents at each FSVM 170, with the contents on different FSVMs 170being synchronized using a distributed data store as described below.For example, the map 360 a may contain entries that map Share-1 andFolder-1 to the File Server FSVM-1 170 a, and Folder-3 to the FileServer FSVM-3 170 c. An example map 360 is shown in Table 1 below.

TABLE 1 Storage item Location Folder-1 FSVM-1 Folder-2 FSVM-1 File-1FSVM-1 Folder-3 FSVM-3 File-2 FSVM-3

In particular embodiments, the map 360 may be accessible on each of thehost machines 201. As described with reference to FIGS. 2A-2B, the maps360 a and 360 c may be copies of a distributed data structure that aremaintained and accessed at each FSVM 170 a-c using a distributed dataaccess coordinator 370 a and 370 c. The distributed data accesscoordinator 370 a and 370 c may be implemented based on distributedlocks or other storage item access operations. Alternatively, thedistributed data access coordinator 370 a and 370 c may be implementedby maintaining a master copy of the maps 360 a and 360 c at a leadernode such as the host machine 201 c, and using distributed locks toaccess the master copy from each FSVM 170 a and 170 b. The distributeddata access coordinator 370 a and 370 c may be implemented usingdistributed locking, leader election, or related features provided by acentralized coordination service for maintaining configurationinformation, naming, providing distributed synchronization, and/orproviding group services (e.g., APACHE ZOOKEEPER or other distributedcoordination software). Since the map 360 a indicates that Folder-3 islocated at FSVM-3 170 c, the lookup operation at step 384 determinesthat Folder-3 is not located at FSVM-1. Thus, at step 385 the FSVM-1 170a sends a response, e.g., a “Not Covered” DFS response, to the client330 indicating that the requested folder is not located at FSVM-1. Atstep 386, the client 330 sends a request to FSVM-1 for a referral to theFSVM on which Folder-3 is located. FSVM-1 uses the map 360 a todetermine that Folder-3 is located at FSVM-3, and at step 387 returns aresponse, e.g., a “Redirect” DFS response, redirecting the client 330 toFSVM-3. The client 330 may then determine the network address forFSVM-3. In one example, the network address may be a host nameconstructed from the FSVM name, e.g., the host name “fsvm-3.domain.name”for FSVM-3. In another example the host name may be associated with theFSVM. The host name for FSVM-3 may be ip-addr3 (e.g., a host name“fsvm-3.domain.name” or an IP address, 10.1.1.3). The client 330 maydetermine the network address for FSVM-3 by searching a cache stored inmemory of the client 330, which may contain a mapping from FSVM-3 toip-addr3 cached in a previous operation. If the cache does not contain anetwork address for FSVM-3, then at step 388 the client 330 may send arequest to the name server 332 to resolve the name FSVM-3. The nameserver may respond with the resolved address, ip-addr3, at step 389. Theclient 330 may then store the association between FSVM-3 and ip-addr3 inthe client's cache.

In particular embodiments, failure of FSVMs 170 may be detected usingthe centralized coordination service. For example, using the centralizedcoordination service, each FSVM 170 a may create a lock on the hostmachine 201 a on which the FSVM 170 a is located using ephemeral nodesof the centralized coordination service (which are different from hostmachines 201 but may correspond to host machines 201). Other FSVMs 170 band 170 c may volunteer for leadership of resources of remote FSVMs 170on other host machines 201, e.g., by requesting a lock on the other hostmachines 201. The locks requested by the other nodes are not grantedunless communication to the leader host machine 201 c is lost, in whichcase the centralized coordination service deletes the ephemeral node andgrants the lock to one of the volunteer host machines 201 a and 201 b,which becomes the new leader. For example, the volunteer host machines201 a and 201 b may be ordered by the time at which the centralizedcoordination service received their requests, and the lock may begranted to the first host machine 201 on the ordered list. The firsthost machine 201 (e.g., host machine 201 b) on the list may thus beselected as the new leader. The FSVM 170 b on the new leader hasownership of the resources that were associated with the failed leaderFSVM 170 a until the failed leader FSVM 170 c is restored, at whichpoint the restored FSVM 170 a may reclaim the local resources of thehost machine 201 c on which it is located.

At step 390, the client 330 may send an access request to FSVM-3 170 cat ip-addr3 requesting the contents of Folder-3 of Share-1. At step 391,FSVM-3 170 c queries FSVM-3's copy of the map 360 using FSVM-3'sinstance of the distributed data access coordinator 370 c. The map 360indicates that Folder-3 is located on FSVM-3, so at step 392 FSVM-3accesses the filesystem 364 c to retrieve information about Folder-3 316and its contents (e.g., a list of files in the folder, which includesFile-2 320) that are stored on the local storage 122 c. FSVM-3 mayaccess local storage 122 c via CVM-3 110 c, which provides access tolocal storage 122 c via a volume group 368 c that contains one or morevolumes stored on one or more storage devices in local storage 122 c. Atstep 393, FSVM-3 may then send the information about Folder-3 and itscontents to the client 330. Optionally, FSVM-3 may retrieve the contentsof File-2 and send them to the client 330, or the client 330 may send asubsequent request to retrieve File-2 as needed.

FIG. 3D illustrates an example virtualized file server having a failovercapability according to particular embodiments. To provide highavailability, e.g., so that the file server continues to operate afterfailure of components such as a CVM, FSVM, or both, as may occur if ahost machine fails, components on other host machines may take over thefunctions of failed components. When a CVM fails, a CVM on another hostmachine may take over input/output operations for the failed CVM.Further, when an FSVM fails, an FSVM on another host machine may takeover the network address and CVM or volume group that were being used bythe failed FSVM. If both an FSVM and an associated CVM on a host machinefail, as may occur when the host machine fails, then the FSVM and CVM onanother host machine may take over for the failed FSVM and CVM. When thefailed FSVM and/or CVM are restored and operational, the restored FSVMand/or CVM may take over the operations that were being performed by theother FSVM and/or CVM. In FIG. 3D, FSVM-1 170 a communicates with CVM-1110 a to use the data storage in volume groups VG1 368 a and VG2 369 a.For example, FSVM-1 is using disks in VG1 and VG2, which are iSCSItargets. FSVM-1 has iSCSI initiators that communicate with the VG1 andVG2 targets using MPIO (e.g., DM-MPIO on the LINUX operating system).FSVM-1 may access the volume groups VG1 and VG2 via in-guest iSCSI.Thus, any FSVM may connect to any iSCSI target if an FSVM failureoccurs.

In particular embodiments, during failure-free operation, there areactive iSCSI paths between FSVM-1 and CVM-1, as shown in FIG. 3D by thedashed lines from the FSVM-1 filesystems for FS1 364 a and FS2 365 a toCVM-1 's volume group VG1 368 a and VG2 369 a, respectively. Further,during failure-free operation there are inactive failover (e.g.,standby) paths between FSVM-1 and CVM-3 110 c, which is located onHost-3. The failover paths may be, e.g., paths that are ready to beactivated in response to the local CVM CVM-1 becoming unavailable. Theremay be additional failover paths that are not shown in FIG. 3D. Forexample, there may be failover paths between FSVM-1 and a CVM on anotherhost machine, such as CVM-2 110 b on Host-2 201 b. The local CVM CVM-1110 a may become unavailable if, for example, CVM-1 crashes, or the hostmachine on which the CVM-1 is located crashes, loses power, losesnetwork communication between FSVM-1 170 a and CVM-1 110 a. As anexample and not by way of limitation, the failover paths do not performI/O operations during failure-free operation. Optionally, metadataassociated with a failed CVM 110 a, e.g., metadata related to volumegroups 368 a, 369 a associated with the failed CVM 110 a, may betransferred to an operational CVM, e.g., CVM 110 c, so that the specificconfiguration and/or state of the failed CVM 110 a may be re-created onthe operational CVM 110 c.

FIG. 3E illustrates an example virtualized file server that hasrecovered from a failure of Controller/Service VM CVM-1 110 a byswitching to an alternate Controller/Service VM CVM-3 110 c according toparticular embodiments. When CVM-1 110 a fails or otherwise becomesunavailable, then the FSVM associated with CVM-1, FSVM-1 170 a, maydetect a PATH DOWN status on one or both of the iSCSI targets for thevolume groups VG1 368 a and VG2 369 a, and initiate failover to a remoteCVM that can provide access to those volume groups VG1 and VG2. Further,if a CVM's host machine crashes, and volume groups such as VG1 and VG2are stored on the crashed host machine, one or more CVMs on one or moreother host machines may provide access to those volume groups (e.g., VG1and VG2) using redundant data and metadata that is stored on the CVMcluster. One or more redundant copies of the data and metadata may bestored, and the number of CVM failures that can be tolerated correspondsto the number of redundant copies. For example, when CVM-1 110 a fails,the iSCSI MPIO may activate failover (e.g., standby) paths to the remoteiSCSI target volume group(s) associated with the remote CVM-3 110 c onHost-3 201 c. CVM-3 provides access to volume groups VG1 and VG2 as VG1368 c and VG2 369 c, which are on storage device(s) of local storage 122c. The activated failover path may take over I/O operations from failedCVM-1 110 a. Optionally, metadata associated with the failed CVM-1 110a, e.g., metadata related to volume groups 368 a, 369 a, may betransferred to CVM-3 so that the specific configuration and/or state ofCVM-1 may be re-created on CVM-3. When the failed CVM-1 again becomesavailable, e.g., after it has been re-started and has resumed operation,the path between FSVM-1 and CVM-1 may reactivated or marked as theactive path, so that local I/O between CVM-1 and FSVM-1 may resume, andthe path between CVM-3 and FSVM-1 may again become a failover (e.g.,standby) path.

FIG. 3F illustrates an example virtualized file server that hasrecovered from failure of a FSVM by electing a new leader FSVM accordingto particular embodiments. When an FSVM-2 170 b fails, e.g., because ithas been brought down for maintenance, has crashed, the host machine onwhich it was executing has been powered off or crashed, networkcommunication between the FSVM and other FSVMs has become inoperative,or other causes, then the CVM that was being used by the failed FSVM,the CVM's associated volume group(s), and the network address of thehost machine on which the failed FSVM was executing may be taken over byanother FSVM to provide continued availability of the file services thatwere being provided by the failed FSVM. In the example shown in FIG. 3F,FSVM-2 170 b on Host-2 201 b has failed. One or more other FSVMs, e.g.,FSVM-1 170 a or FSVM-3 170 c, or other components located on one or moreother host machines, may detect the failure of FSVM-2, e.g., bydetecting a communication timeout or lack of response to a periodicstatus check message. When FSVM-2 's failure is detected, an electionmay be held, e.g., using a distributed leader election process such asthat provided by the centralized coordination service. The host machinethat wins the election may become the new leader for the filesystempools 366 b, 367 b for which the failed FSVM-2 was the leader. In thisexample, FSVM-1 170 a wins the election and becomes the new leader forthe pools 366 b, 367 b. FSVM-1 170 a thus attaches to CVM-2 110 b bycreating filesystem 364 b, 365 b instances for the file server instancesFS1 and FS2 using FS1-Pool-3 366 b and FS2-Pool-4 367 b, respectively.In this way, FSVM-1 takes over the filesystems and pools for CVM-2 'svolume groups, e.g., volume groups VG1 366 b and VG2 367 b of localstorage 122 b. Further, FSVM-1 takes over the IP address associated withFSVM-2, 10.1.1.2, so that storage access requests sent to FSVM-2 arereceived and processed by FSVM-1. Host-2 201 b may continue to operate,in which case CVM-2 110 b may continue to execute on Host-2. When FSVM-2again becomes available, e.g., after it has been re-started and hasresumed operation, FSVM-2 may assert leadership and take back its IPaddress (10.1.1.2) and storage (FS1-Pool-3 366 b and FS2-Pool-4 367 b)from FSVM-1.

FIGS. 3G and 311 illustrate example virtualized file servers that haverecovered from failure of a host machine 201 a by switching to anotherController/Service VM and another FSVM according to particularembodiments. The other Controller/Service VM and FSVM are located on asingle host machine 201 c in FIG. 3G, and on two different host machines201 b, 201 c in FIG. 3H. In both FIGS. 3G and 3H, Host-1 201 a hasfailed, e.g., crashed or otherwise become inoperative or unresponsive tonetwork communication. Both FSVM-1 170 a and CVM-1 110 a located on thefailed Host-1 201 a have thus failed. Note that the CVM 110 a and FSVM170 a on a particular host machine 201 a may both fail even if the hostmachine 201 a itself does not fail. Recovery from failure of a CVM 110 aand an FSVM 170 a located on the same host machine 201 a, regardless ofwhether the host machine 201 a itself failed, may be performed asfollows. The failure of FSVM-1 and CVM-1 may be detected by one or moreother FSVMs, e.g., FSVM-2 170 b, FSVM-3 170 c, or by other componentslocated on one or more other host machines. FSVM-1 's failure may bedetected when a communication timeout occurs or there is no response toa periodic status check message within a timeout period, for example.CVM-1 's failure may be detected when a PATH DOWN condition occurs onone or more of CVM-1 's volume groups' targets (e.g., iSCSI targets).

When FSVM-1 's failure is detected, an election may be held as describedabove with reference to FIG. 3F to elect an active FSVM to take overleadership of the portions of the file server instance for which thefailed FSVM was the leader. These portions are FileSystem-1A 364 a forthe portion of file server FS1 located on FSVM-1, and FileSystem-2A 365a for the portion of file server FS2 located on FSVM-1. FileSystem-1A364 a uses the pool FS-Pool-1 366 a and FileSystem-2A 365 a uses thepool FS2-Pool-2 367 a. Thus, the FileSystem-1A 364 a and FileSystem-2Amay be re-mounted on the new leader FSVM-3 170 c on Host-3 201 c.Further, FSVM-3 170 c may take over the IP address associated withfailed FSVM-1 170 a, 10.1.1.1, so that storage access requests sent toFSVM-1 are received and processed by FSVM-3.

One or more failover paths from an FSVM to volume groups on one or moreCVMs may be defined for use when a CVM fails. When CVM-1 's failure isdetected, the MPIO may activate one of the failover (e.g., standby)paths to remote iSCSI target volume group(s) associated with a remoteCVM. For example, there may be a first predefined failover path fromFSVM-1 to the volume groups VG1 368 c, 369 c in CVM-3 (which are on thesame host as FSVM-1 when FSVM-1 is restored on Host-3 in examples ofFIGS. 3G and 3H), and a second predefined failover path to the volumegroups VG1 368 b, VG2 369 b in CVM-2. The first failover path, to CVM-3,is shown in FIG. 3G, and the second failover path, to CVM-2 is shown inFIG. 3H. An FSVM or MPIO may choose the first or second failover pathaccording to the predetermined MPIO failover configuration that has beenspecified by a system administrator or user. The failover configurationmay indicate that the path is selected (a) by reverting to the previousprimary path, (b) in order of most preferred path, (c) in a round-robinorder, (d) to the path with the least number of outstanding requests,(e) to the path with the least weight, or (f) to the path with the leastnumber of pending requests. When failure of CVM-1 110 a is detected,e.g., by FSVM-1 or MPIO detecting a PATH DOWN condition on one of CVM-1's volume groups VG1 368 a or VG2 369 a, the alternate CVM on theselected failover path may take over I/O operations from the failedCVM-1. As shown in FIG. 3G, if the first failover path is chosen, CVM-3110 c on Host-3 201 c is the alternate CVM, and the pools FS1-Pool-1 366a and FS2-Pool-2 367 a, used by the filesystems FileSystem-1A 364 a andFileSystem-2A 365 a, respectively, which have been restored on FSVM-3 onHost-3, may use volume groups VG1 368 c and VG2 369 c of CVM-3 110 c onHost-3 when the first failover path is chosen. Alternatively, as shownin FIG. 3H, if the second failover path is chosen, CVM-2 on Host-2 isthe alternate CVM, and the pools FS1-Pool-1 366 a and FS2-Pool-2 367 aused by the respective filesystems FileSystem-1A 364 a and FileSystem-2A365 a, which have been restored on FSVM-3, may use volume groups VG1 368b and VG2 369 b on Host-2, respectively.

In particular embodiments, metadata associated with the failed CVM-1 110a, e.g., metadata related to volume groups 368 a, 369 a, may betransferred to the alternate CVM (e.g., CVM-2 or CVM-3) that thespecific configuration and/or state of CVM-1 may be re-created on thealternative CVM. When FSVM-1 again becomes available, e.g., after it hasbeen re-started and has resumed operation on Host-1 201 a or anotherhost machine, FSVM-1 may assert leadership and take back its IP address(10.1.1.1) and storage assignments (FileSystem-1A and FS1-Pool-1 366 a,and FileSystem-2A and FS2-Pool-2 366 b) from FSVM-3. When CVM-1 againbecomes available, MPIO or FSVM-1 may switch the FSVM to CVMcommunication paths (iSCSI paths) for FileSystem-1A 364 a andFileSystem-2A 365 a back to the pre-failure paths, e.g., the paths tovolume groups VG1 368 a and 369 a in CVM-1 110 a, or the selectedalternate path may remain in use. For example, the MPIO configurationmay specify that fail back to FSVM-1 is to occur when the primary pathis restored, since communication between FSVM-1 and CVM-1 is local andmay be faster than communication between FSVM-1 and CVM-2 or CVM-3. Inthis case, the paths between CVM-2 and/or CVM-3 and FSVM-1 may againbecome failover (e.g., standby) paths.

FIGS. 4A and 4B illustrate an example hierarchical namespace 400 of afile server according to particular embodiments. Cluster-1 402 is acluster, which may contain one or more file server instances, such as aninstance named FS1.domain.com 404. Although one cluster is shown inFIGS. 4A and 4B, there may be multiple clusters, and each cluster mayinclude one or more file server instances. The file serverFS1.domain.com 404 contains three shares: Share-1 406, Share-2 408, andShare-3 410. Share-1 may be a home directory share on which userdirectories are stored, and Share-2 and Share-3 may be departmentalshares for two different departments of a business organization, forexample. Each share has an associated size in gigabytes, e.g., 100 Gb(gigabytes) for Share-1, 100 Gb for Share-2, and 10 Gb for Share-3. Thesizes may indicate a total capacity, including used and free space, ormay indicate used space or free space. Share-1 includes three folders,Folder-A1 412, Folder-A2 414, and Folder-A3 416. The capacity ofFolder-A1 is 18 Gb, Folder-A2 is 16 Gb, and Folder-A3 is 66 Gb. Further,each folder is associated with a user, referred to as an owner.Folder-A1 is owned by User-1, Folder-A2 by User-2, and Folder-A3 byUser-3. Folder-A1 contains a file named File-A1-1 418, of size 18 Gb.Folder-A2 contains 32 files, each of size 0.5 Gb, named File-A2-1 420through File-A2-32 422. Folder-A3 contains 33 files, each of size 2 Gb,named File-A3-1 423 and File-A3-2 424 through File-A3-33 426.

FIG. 4B shows the contents of Share-2 408 and Share-3 410 ofFS1.domain.com 404. Share-2 contains a folder named Folder-B1 440, ownedby User-1 and having a size of 100 Gb. Folder-B1 contains File-B1-1 442of size 20 Gb, File-B1-2 444 of size 30 Gb, and Folder-B2 446, owned byUser-2 and having size 50 Gb. Folder-B2 contains File-B2-1 448 of size 5Gb, File-B2-2 450 of size 5 Gb, and Folder-B3 452, owned by User-3 andhaving size 40 Gb. Folder-B3 452 contains 20 files of size 2 Gb each,named File-B3-1 454 through File-B3-20 456. Share-3 contains threefolders: Folder-C7 429 owned by User-1 of size 3 Gb, Folder-C8 430 ownedby User-2 of size 3 Gb, and Folder-C9 432 owned by User-3 of size 4 Gb.

FIG. 4C illustrates distribution of stored data amongst host machines ina virtualized file server according to particular embodiments. In theexample of FIG. 4C, the three shares are spread across three hostmachines 201 a-c. Approximately one-third of each share is located oneach of the three FSVMs 170 a-c. For example, approximately one-third ofShare-3's files are located on each of the three FSVMs 170 a-c. Notethat from a user's point of a view, a share looks like a directory.Although the files in the shares (and in directories) are distributedacross the three FSVMs 170 a-c, the VFS 202 provides a directorystructure having a single namespace in which client executing on userVMs 105 may access the files in a location-transparent way, e.g.,without knowing which FSVMs store which files (or which blocks offiles).

In the example of FIG. 4C, Host-1 stores (e.g., is assigned to) 28 Gb ofShare-1, including 18 Gb for File-A1-1 418 and 2 Gb each for File-A3-1423 through File-A3-5 425, 33 Gb of Share-2, including 20 Gb forFile-B1-1 and 13 Gb for File-B1-2, and 3 Gb of Share-3, including 3 Gbof Folder-C7. Host-2 stores 26 Gb of Share-1, including 0.5 Gb each ofFile-A2-1 420 through File-A2-32 422 (16 Gb total) and 2 Gb each ofFile-A3-6 426 through File-A3-10 427 (10 Gb total), 27 Gb of Share-2,including 17 Gb of File-B1-2, 5 Gb of File-B2-1, and 5 Gb of File-B 2-2,and 3 Gb of Share-3, including 3 Gb of Folder-C8. Host-3 stores 46 Gb ofShare-1, including 2 Gb each of File-A3-11 429 through File-A3-33 428(66 Gb total), 40 Gb of Share-2, including 2 Gb each of File-B3-1 454through File-B3-20 456, and Share-3 stores 4 Gb of Share-3, including 4Gb of Folder-C9 432.

In particular embodiments, a system for managing communicationconnections in a virtualization environment includes a plurality of hostmachines implementing a virtualization environment. Each of the hostmachines includes a hypervisor and at least one user VM 105. The systemmay also include a connection agent, an I/O controller, and/or a virtualdisk comprising a plurality of storage devices. The virtual disk may beaccessible by all of the I/O controllers, and the I/O controllers mayconduct I/O transactions with the virtual disk based on I/O requestsreceived from the user VMs 105. The I/O requests may be, for example,requests to perform particular storage access operations such as listfolders and files in a specified folder, create a new file or folder,open an existing file for reading or writing, read data from or writedata to a file, as well as file manipulation operations to rename,delete, copy, or get details, such as metadata, of files or folders.Each I/O request may reference, e.g., identify by name or numericidentifier, a file or folder on which the associated storage accessoperation is to be performed. The system further includes a virtualizedfile server, which includes a plurality of FSVMs 170 and associatedlocal storage 122. Each FSVM 170 and associated local storage device 122is local to a corresponding one of the host machines 201. The FSVMs 170conduct I/O transactions with their associated local storage 122 basedon I/O requests received from the user VMs 105. For each one of the hostmachines 201, each of the user VMs 105 on the one of the host machines201 sends each of its respective I/O requests 383 to a selected one ofthe FSVMs 170, which may be selected based on a lookup table 360, e.g.,a sharding map, that maps a file 318, folder 312, or other storageresource referenced by the I/O request to the selected one of the FSVMs170).

In particular embodiments, the initial FSVM to receive the request fromthe user VM may be determined by selecting any of the FSVMs 170 on thenetwork 140, e.g., at random, by round robin selection, or by aload-balancing algorithm, and sending an I/O request 383 to the selectedFSVM 170 via the network 140 or via local communication within the hostmachine 201. Local communication may be used if the file 318 or folder412 referenced by the I/O request is local to the selected FSVM, e.g.,the referenced file or folder is located on the same host machine 201 asthe selected FSVM 170. In this local case, the I/O request 383 need notbe sent via the network 140. Instead, the I/O request 383 may be sent tothe selected FSVM 170 using local communication, e.g., a localcommunication protocol such as UNIX domain sockets, a loopbackcommunication interface, inter-process communication on the host machine201, or the like. The selected FSVM 170 may perform the I/O transactionspecified in the I/O request and return the result of the transactionvia local communication. If the referenced file or folder is not localto the selected FSVM, then the selected FSVM may return a resultindicating that the I/O request cannot be performed because the file orfolder is not local to the FSVM. The user VM may then submit a REFERRALrequest or the like to the selected FSVM, which may determine which FSVMthe referenced file or folder is local to (e.g., by looking up the FSVMin a distributed mapping table), and return the identity of that FSVM tothe user VM in a REDIRECT response or the like. Alternatively, theselected FSVM may determine which FSVM the referenced file or folder islocal to, and return the identity of that FSVM to the user VM in thefirst response without the REFERRAL and REDIRECT messages. Other ways ofredirecting the user VM to the FSVM of the referenced file arecontemplated. For example, the FSVM that is on the same host as therequesting user VM (e.g., local to the requesting user VM) may determinewhich FSVM the file or folder is local to, and inform the requestinguser VM of the identity of that FSVM without communicating with adifferent host.

In particular embodiments, the file or folder referenced by the I/Orequest includes a file server name that identifies a virtualized fileserver on which the file or folder is stored. The file server name mayalso include or be associated with a share name that identifies a share,filesystem, partition, or volume on which the file or folder is stored.Each of the user VMs on the host machine may send a host name lookuprequest, e.g., to a domain name service, that includes the file servername, and may receive one or more network addresses of one or more hostmachines on which the file or folder is stored.

In particular embodiments, as described above, the FSVM may send the I/Orequest to a selected one of the FSVMs. The selected one of the FSVMsmay be identified by one of the host machine network addresses receivedabove. In one aspect, the file or folder is stored in the local storageof one of the host machines, and the identity of the host machines maybe determined as described below.

In particular embodiments, when the file or folder is not located onstorage local to the selected FSVM, e.g., when the selected FSVM is notlocal to the identified host machine, the selected FSVM responds to theI/O request with an indication that the file or folder is not located onthe identified host machine. Alternatively, the FSVM may look up theidentity of the host machine on which the file or folder is located, andreturn the identity of the host machine in a response.

In particular embodiments, when the host machine receives a responseindicating that the file or folder is not located in the local storageof the selected FSVM, the host machine may send a referral request(referencing the I/O request or the file or folder from the I/O request)to the selected FSVM. When the selected FSVM receives the referralrequest, the selected FSVM identifies one of the FSVMs 170 that isassociated with a file or folder referenced in the referral requestbased on an association that maps files to FSVMs, such as a shardingtable (which may be stored by the centralized coordination service).When the selected FSVM is not local to the host machine, then theselected FSVM sends a redirect response that redirects the user VM onthe host machine to the selected FSVM. That is, the redirect responsemay reference the identified FSVM. In particular embodiments, the userVM on the host machine receives the redirect response and may cache anassociation between the file or folder referenced in the I/O request andthe FSVM 170 referenced in the redirect response.

In particular embodiments, the user VM on the host machine may send ahost name lookup request that includes the name of the identified FSVM170 to a name service, and may receive the network address of theidentified FSVM from the name service. The user VM on the host machinemay then send the I/O request to the network address received from thename service. The FSVM on the host machine may receive the I/O requestand performs the I/O transaction specified therein. That is, when theFSVM is local to the identified host machine, the FSVM performs the I/Otransaction based on the I/O request. After performing or requesting theI/O transaction, the FSVM may send a response that includes a result ofthe I/O transaction back to the requesting host machine. I/O requestsfrom the user VM may be generated by a client library that implementsfile I/O and is used by client program code (such as an applicationprogram).

FIG. 5 illustrates an example method for accessing data in a virtualizedfile server according to particular embodiments. The client system 330may access the data, such as a specified folder, as follows. At step502, the client system 330 receives a storage access request from anapplication executing in a user VM. Each storage access requestreferences a file path (e.g., \\FS1.domain.com:\share-1 \Folder-1),which includes a file or folder name and further includes or can be usedto identify a share name (e.g., FS1.domain.com\share-1) or an NFS remotefilesystem name (e.g., fs1.domain.com:/share-1. The storage accessrequest may also include an operation type (e.g., read, write, delete,rename, etc.), a position in the file (for read/write requests), data tobe written (for write requests), quantity of data to be read (for readrequests), a new file path (for rename requests), folder name (forfolder creation requests) or other information appropriate for theoperation type. At step 504, the client system may send a DNS queryrequest for the file server portion of the share name (e.g.,\\fs1.domain.com for the share \\FS1.domain.com\share-1) to a nameserver 332, which may return the identity of a selected FSVM as aresult. The name server 332 may be a DNS server. The selected FSVM isnot necessarily the FSVM on which the file or folder itself is located,however, since the share may be distributed amongst multiple FSVMs, oneof which actually stores the file or folder. In particular embodiments,a FSVM can determine which FSVM a file is stored on, and, if a FSVMreceives a request for a file stored on a different FSVM, the FSVM sendsa referral response that includes the identity of the FSVM on which thefile is stored.

At step 506, the name server 332 may respond to the client with an IP(network) address of one or more FSVMs where the file or folder may belocated. For example, the DNS server entry FS1.domain.com includesentries for FSVM-1, FSVM-2, and FSVM-3, which are, respectively,ip-addr1, ip-addr2, ip-addr3 (or 10.1.1.1, 10.1.1.2, 10.1.1.3). One ofthese three example IP addresses may be selected by the DNS server andreturned in a response. In one example, the DNS server returns the threeIP addresses in a different permutation for each request using DNS roundrobin so that a different server may be selected by the client for eachrequest to balance the request load among the three servers. In thisexample, ip-addr1 (10.1.1.1) is the first address in the list sent inthe reply to the client 330, and so is selected by the client as theaddress to which the I/O request will, at least initially, be sent. Atstep 508, the client may send the I/O request to access the folder“Folder-3” to the FSVM located on the host machine having addressip-addr1. The I/O request may be, e.g., a DFS attach or connect request,an NFS open request, or the like.

At step 510, FSVM-1 170 a on Host-1 201 a receives the I/O request andconsults a map or lookup table, such as the sharding map 360 a, todetermine whether the folder “Folder-3” is stored on FSVM-1 170 a, e.g.,on a locally-attached storage resource of the host machine on which FSVM170 a is located. If so, FSVM 170 a performs executes step 567 toperform the I/O transaction identified by the I/O request. If not, atstep 512 FSVM-1 170 a responds to the client 330 with an indication thatthe folder is not located on the FSVM-1 170 a. The indication may be,e.g., a PATH NOT COVERED DFS response. At step 514, upon receiving theindication that the file is not located on the FSVM 170 a to which therequest was sent, the client 330 sends a DFS REFERRAL request to FSVM170 a, requesting a referral to the FSVM on which “Folder-3” is stored.At step 545, FSVM 170 a receives the REFERRAL request and sends a DFS“REDIRECT to FSVM-3” response back to the client 330. FSVM 170 a looksup the FSVM on which the folder “Folder-3” is stored in the map 360 athat associates files or shares with FSVMs. The result of the lookup,FSVM-3 170 c, may have been determined previously by the lookup at step510 when the initial request for Folder-3 was received, or may bedetermined at step 516 when the referral request for Folder-3 isreceived. For example, the map 360 a may be stored in a shared datastructure provided by the centralized coordination service, and thelookup may be performed by accessing the shared data structure. In thisexample, the file or folder is “Folder-3” and map indicates that thefolder is associated with FSVM 170 c, so at step 516 FSVM 170 a may senda REDIRECT response to the client indicating that the requested folderis stored on FSVM 170 c. The REDIRECT response may reference the FSVM170 c, the network address of FSVM 170 c (e.g., ip-addr3, in which casesteps 518 and 520 may not be necessary), or other identifier for thelocation of the requested folder. The client 330 may receive theREDIRECT response and cache the association between Folder-3 and FSVM170 c for potential future use.

At step 518, the client 330 may send a DNS query request to the DNSserver 332 to determine the IP address of the FSVM specified in thereceived REDIRECT response, which is FSVM 170 c having IP addressip-addr3 in this example. At step 520, the DNS server 332 may send areply to the client 330 indicating the IP address of the requested FSVM.For example, the reply may be ip-addr3 (or 10.1.1.3), which is the IPaddress of FSVM 170 c. At step 522, the client sends the I/O request toaccess Folder-3 to the IP address received in the DNS reply (e.g.,ip-addr3). At step 524, the FSVM 170 c on host machine 201 c receivesthe I/O request that references Folder-3 and looks up Folder-3 in thesharding map. At step 526, FSVM 170 c performs the requested I/Otransaction for Folder-3, e.g., by accessing local storage 122 c, andsends the results of the access, e.g., details about Folder-3 in thisexample, such as a list of files and associated metadata, back to theclient 330 in an I/O response. The client 330 receives the I/O responseand may pass the results of the I/O transaction to the application orother program code that requested the access. Any subsequent requestsfor the same data (Folder-3 in this example) by the client 330 may besent directly to the FSVM 170 c on which the data is stored because theclient 330 may use the cached identity of the FSVM 170 c on which thedata is stored. Although data contained in a folder is accessed in theexample of FIG. 5, other types of data may be accessed similarly, e.g.,data contained in files.

FIG. 7 is a block diagram of an illustrative computing system 700suitable for implementing particular embodiments. In particularembodiments, one or more computer systems 700 perform one or more stepsof one or more methods described or illustrated herein. In particularembodiments, one or more computer systems 700 provide functionalitydescribed or illustrated herein. In particular embodiments, softwarerunning on one or more computer systems 700 performs one or more stepsof one or more methods described or illustrated herein or providesfunctionality described or illustrated herein. Particular embodimentsinclude one or more portions of one or more computer systems 700.Herein, reference to a computer system may encompass a computing device,and vice versa, where appropriate. Moreover, reference to a computersystem may encompass one or more computer systems, where appropriate.

This disclosure contemplates any suitable number of computer systems700. This disclosure contemplates computer system 700 taking anysuitable physical form. As example and not by way of limitation,computer system 700 may be an embedded computer system, a system-on-chip(SOC), a single-board computer system (SBC) (such as, for example, acomputer-on-module (COM) or system-on-module (SOM)), a desktop computersystem, a mainframe, a mesh of computer systems, a server, a laptop ornotebook computer system, a tablet computer system, or a combination oftwo or more of these. Where appropriate, computer system 700 may includeone or more computer systems 700; be unitary or distributed; spanmultiple locations; span multiple machines; span multiple data centers;or reside in a cloud, which may include one or more cloud components inone or more networks. Where appropriate, one or more computer systems700 may perform without substantial spatial or temporal limitation oneor more steps of one or more methods described or illustrated herein. Asan example and not by way of limitation, one or more computer systems700 may perform in real time or in batch mode one or more steps of oneor more methods described or illustrated herein. One or more computersystems 700 may perform at different times or at different locations oneor more steps of one or more methods described or illustrated herein,where appropriate.

Computer system 700 includes a bus 702 (e.g., an address bus and a databus) or other communication mechanism for communicating information,which interconnects subsystems and devices, such as processor 704,memory 706 (e.g., RAM), static storage 708 (e.g., ROM), dynamic storage710 (e.g., magnetic or optical), communication interface 714 (e.g.,modem, Ethernet card, a network interface controller (NIC) or networkadapter for communicating with an Ethernet or other wire-based network,a wireless NIC (WNIC) or wireless adapter for communicating with awireless network, such as a WI-FI network), input/output (I/O) interface712 (e.g., keyboard, keypad, mouse, microphone). In particularembodiments, computer system 700 may include one or more of any suchcomponents.

In particular embodiments, processor 704 includes hardware for executinginstructions, such as those making up a computer program. As an exampleand not by way of limitation, to execute instructions, processor 704 mayretrieve (or fetch) the instructions from an internal register, aninternal cache, memory 706, static storage 708, or dynamic storage 710;decode and execute them; and then write one or more results to aninternal register, an internal cache, memory 706, static storage 708, ordynamic storage 710. In particular embodiments, processor 704 mayinclude one or more internal caches for data, instructions, oraddresses. This disclosure contemplates processor 704 including anysuitable number of any suitable internal caches, where appropriate. Asan example and not by way of limitation, processor 704 may include oneor more instruction caches, one or more data caches, and one or moretranslation lookaside buffers (TLBs). Instructions in the instructioncaches may be copies of instructions in memory 706, static storage 708,or dynamic storage 710, and the instruction caches may speed upretrieval of those instructions by processor 704. Data in the datacaches may be copies of data in memory 706, static storage 708, ordynamic storage 710 for instructions executing at processor 704 tooperate on; the results of previous instructions executed at processor704 for access by subsequent instructions executing at processor 704 orfor writing to memory 706, static storage 708, or dynamic storage 710;or other suitable data. The data caches may speed up read or writeoperations by processor 704. The TLBs may speed up virtual-addresstranslation for processor 704. In particular embodiments, processor 704may include one or more internal registers for data, instructions, oraddresses. This disclosure contemplates processor 704 including anysuitable number of any suitable internal registers, where appropriate.Where appropriate, processor 704 may include one or more arithmeticlogic units (ALUs); be a multi-core processor; or include one or moreprocessors 702. Although this disclosure describes and illustrates aparticular processor, this disclosure contemplates any suitableprocessor.

In particular embodiments, I/O interface 712 includes hardware,software, or both, providing one or more interfaces for communicationbetween computer system 700 and one or more I/O devices. Computer system700 may include one or more of these I/O devices, where appropriate. Oneor more of these I/O devices may enable communication between a personand computer system 700. As an example and not by way of limitation, anI/O device may include a keyboard, keypad, microphone, monitor, mouse,printer, scanner, speaker, still camera, stylus, tablet, touch screen,trackball, video camera, another suitable I/O device or a combination oftwo or more of these. An I/O device may include one or more sensors.This disclosure contemplates any suitable I/O devices and any suitableI/O interfaces 712 for them. Where appropriate, I/O interface 712 mayinclude one or more device or software drivers enabling processor 704 todrive one or more of these I/O devices. I/O interface 712 may includeone or more I/O interfaces 712, where appropriate. Although thisdisclosure describes and illustrates a particular I/O interface, thisdisclosure contemplates any suitable I/O interface.

In particular embodiments, communication interface 714 includeshardware, software, or both providing one or more interfaces forcommunication (such as, for example, packet-based communication) betweencomputer system 700 and one or more other computer systems 700 or one ormore networks. As an example and not by way of limitation, communicationinterface 714 may include a network interface controller (NIC) ornetwork adapter for communicating with an Ethernet or other wire-basednetwork or a wireless NIC (WNIC) or wireless adapter for communicatingwith a wireless network, such as a WI-FI network. This disclosurecontemplates any suitable network and any suitable communicationinterface 714 for it. As an example and not by way of limitation,computer system 700 may communicate with an ad hoc network, a personalarea network (PAN), a local area network (LAN), a wide area network(WAN), a metropolitan area network (MAN), or one or more portions of theInternet or a combination of two or more of these. One or more portionsof one or more of these networks may be wired or wireless. As anexample, computer system 700 may communicate with a wireless PAN (WPAN)(such as, for example, a BLUETOOTH WPAN), a WI-FI network, a WI-MAXnetwork, a cellular telephone network (such as, for example, a GlobalSystem for Mobile Communications (GSM) network), or other suitablewireless network or a combination of two or more of these. Computersystem 700 may include any suitable communication interface 714 for anyof these networks, where appropriate. Communication interface 714 mayinclude one or more communication interfaces 714, where appropriate.Although this disclosure describes and illustrates a particularcommunication interface, this disclosure contemplates any suitablecommunication interface.

One or more memory buses (which may each include an address bus and adata bus) may couple processor 704 to memory 706. Bus 702 may includeone or more memory buses, as described below. In particular embodiments,one or more memory management units (MMUs) reside between processor 704and memory 706 and facilitate accesses to memory 706 requested byprocessor 704. In particular embodiments, memory 706 includes randomaccess memory (RAM). This RAM may be volatile memory, where appropriate.Where appropriate, this RAM may be dynamic RAM (DRAM) or static RAM(SRAM). Moreover, where appropriate, this RAM may be single-ported ormulti-ported RAM. This disclosure contemplates any suitable RAM. Memory706 may include one or more memories 706, where appropriate. Althoughthis disclosure describes and illustrates particular memory, thisdisclosure contemplates any suitable memory.

Where appropriate, the ROM may be mask-programmed ROM, programmable ROM(PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM),electrically alterable ROM (EAROM), or flash memory or a combination oftwo or more of these. In particular embodiments, dynamic storage 710 mayinclude a hard disk drive (HDD), a floppy disk drive, flash memory, anoptical disc, a magneto-optical disc, magnetic tape, or a UniversalSerial Bus (USB) drive or a combination of two or more of these. Dynamicstorage 710 may include removable or non-removable (or fixed) media,where appropriate. Dynamic storage 710 may be internal or external tocomputer system 700, where appropriate. This disclosure contemplatesmass dynamic storage 710 taking any suitable physical form. Dynamicstorage 710 may include one or more storage control units facilitatingcommunication between processor 704 and dynamic storage 710, whereappropriate.

In particular embodiments, bus 702 includes hardware, software, or bothcoupling components of computer system 700 to each other. As an exampleand not by way of limitation, bus 702 may include an AcceleratedGraphics Port (AGP) or other graphics bus, an Enhanced Industry StandardArchitecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT)interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBANDinterconnect, a low-pin-count (LPC) bus, a memory bus, a Micro ChannelArchitecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, aPCI-Express (PCIe) bus, a serial advanced technology attachment (SATA)bus, a Video Electronics Standards Association local (VLB) bus, oranother suitable bus or a combination of two or more of these. Bus 702may include one or more buses 706, where appropriate. Although thisdisclosure describes and illustrates a particular bus, this disclosurecontemplates any suitable bus or interconnect.

According to particular embodiments, computer system 700 performsspecific operations by processor 704 executing one or more sequences ofone or more instructions contained in memory 706. Such instructions maybe read into memory 706 from another computer readable/usable medium,such as static storage 708 or dynamic storage 710. In alternativeembodiments, hard-wired circuitry may be used in place of or incombination with software instructions. Thus, particular embodiments arenot limited to any specific combination of hardware circuitry and/orsoftware. In one embodiment, the term “logic” shall mean any combinationof software or hardware that is used to implement all or part ofparticular embodiments disclosed herein.

The term “computer readable medium” or “computer usable medium” as usedherein refers to any medium that participates in providing instructionsto processor 704 for execution. Such a medium may take many forms,including but not limited to, nonvolatile media and volatile media.Non-volatile media includes, for example, optical or magnetic disks,such as static storage 708 or dynamic storage 710. Volatile mediaincludes dynamic memory, such as memory 706.

Common forms of computer readable media include, for example, floppydisk, flexible disk, hard disk, magnetic tape, any other magneticmedium, CD-ROM, any other optical medium, punch cards, paper tape, anyother physical medium with patterns of holes, RAM, PROM, EPROM,FLASH-EPROM, any other memory chip or cartridge, or any other mediumfrom which a computer can read.

In particular embodiments, execution of the sequences of instructions isperformed by a single computer system 700. According to other particularembodiments, two or more computer systems 700 coupled by communicationlink 716 (e.g., LAN, PTSN, or wireless network) may perform the sequenceof instructions in coordination with one another.

Computer system 700 may transmit and receive messages, data, andinstructions, including program, i.e., application code, throughcommunication link 716 and communication interface 714. Received programcode may be executed by processor 704 as it is received, and/or storedin static storage 708 or dynamic storage 710, or other non-volatilestorage for later execution. A database 720 may be used to store dataaccessible by the system 700 by way of data interface 718.

Herein, a computer-readable non-transitory storage medium or media mayinclude one or more semiconductor-based or other integrated circuits(ICs) (such, as for example, field-programmable gate arrays (FPGAs) orapplication-specific ICs (ASICs)), hard disk drives (HDDs), hybrid harddrives (HHDs), optical discs, optical disc drives (ODDs),magneto-optical discs, magneto-optical drives, floppy diskettes, floppydisk drives (FDDs), magnetic tapes, solid-state drives (SSDs),RAM-drives, SECURE DIGITAL cards or drives, any other suitablecomputer-readable non-transitory storage media, or any suitablecombination of two or more of these, where appropriate. Acomputer-readable non-transitory storage medium may be volatile,non-volatile, or a combination of volatile and non-volatile, whereappropriate.

In particular embodiments, a deployment system may deploy, e.g., installand configure components of, a VFS 202 on host machines 202. Thedeployment system may be implemented by an infrastructure managementservice 222. The infrastructure management service 222 may alsoimplement at least a portion of an administrative user interface, whichmay be used by a system administrator or other user to interact with thedeployment system. For example, the administrative user interface may bepresented on a display of a client system 201 or other computing devicethat communicates with the infrastructure management service 222 via thenetwork 140. The infrastructure management service 222 may interact withthe components of the VFS 202 as appropriate, such as the CVM 110, toperform operations on individual host machines 201.

In particular embodiments, the deployment system includes anadministrative user interface that provides a deployment command to beinvoked by an administrator to create a VFS 202. When the deploymentcommand is invoked, the deployment system may perform one or moredeployment operations to deploy the VFS 202. The deployment operationsmay have one or more deployment parameters that specify aspects of thedeployment. The administrative user interface may request one or moredeployment parameters from the system administrator and perform thedeployment operations when the deployment parameters have been received.In one example, the requested deployment parameters may include a namefor the VFS 202, how many FSVMs 170 to include in the VFS 202 (e.g., thenumber of host machines 201 to which the VFS 202 is to be deployed), theamount of storage capacity to allocate to the VFS 202, network details,such as addresses of internal and external networks, and name servicedetails, such as an ACTIVE DIRECTORY name, user name, and accesscredentials such as passwords. In particular embodiments, deploymentparameters, such as one or more of the aforementioned parameters orother parameters not described in this example, may be determinedautomatically without being requested from the user.

A described above with reference to FIG. 2A, a VFS 202 may includemultiple compute units, e.g., FSVMs 170 located on host machines 201.These FSVMs 170 may act as a single VFS 202 to the outside world. A VFS202 may be a cluster of at least three File Server Virtual Machines(FSVMs) 170 a-c located on three respective host machines 201 a-c. Toprovide high-availability, there may be a maximum of one FSVM 170 a fora particular VFS instance 202 per host machine 201 in a cluster. In oneexample, the VFS 202 may be deployed on host machines 201 on which oneor more of the components shown in FIG. 1A, such as the user VM 105, theCVM 110 a, the hypervisor 130, are installed and operational. In anotherexample, the VFS 202 may be deployed on host machines 201 that have notbeen configured, such as host machines 201 that are being newly-added oron which no software, e.g., none of the components shown in in FIG. 1A,is installed.

In particular embodiments, a deployment image that contains program codeinstructions and data is provided to each host machine 201 of the VFS202. The deployment image may be, e.g., a disk image containing computerprogram code, such as operating system code that implements an operatingsystem (e.g., LINUX, MICROSOFT WINDOWS, or the like), code thatimplements one or more of the components shown in FIG. 1A, such as theuser VM 105, CVM 110, hypervisor 130, or the like. The deployment imagemay contain program code instructions and data for the FSVM 170, whichmay implement features of the VFS 202 on each host machine 201. When thedeployment image becomes available to each host machine, the hostmachine may boot from the deployment image and begin executing a user VM105, CVM 110, and FSVM 170. Clusters of one or more host machines 201having appropriate platforms (e.g., hardware configurations) andlicenses (e.g., operating system and application licenses) may use ahypervisor-agnostic deployment image. Each host machine 201 may bootfrom a deployment image by, for example, loading and executing theprogram code instructions in the deployment image. As a result, the FSVM170 and related components shown in FIG. 2A execute on each host machine201 and provide at least a portion of the VFS 202. A set of one or moresuch deployment images may be pre-created and accessed as needed, e.g.,when a deployment command is issued or when a host machine begins toboot and load the deployment image.

In particular embodiments, there may be separate boot and data disks,which may be included in a single deployment image or may be in separatedeployment images. Program code instructions such as the operatingsystem (OS) code and FSVM code may be stored on the boot vdisk. Thefileserver persistent data and configuration may be stored on the datavdisk. The FSVM 170 on each host machine 201 may be at least partiallyconfigured at runtime, e.g., with data that is specific to eachparticular instance of an FSVM on each corresponding host machine 201.User data, such as user filesystems, may be stored separately from thedeployment image(s), and may be accessed by each FSVM 170 when the FSVM170 is ready to provide its portion of the VFS (e.g., when the FSVM 170has finished booting).

As introduced above, an administrative user interface may provide adeployment command that an administrator can invoke to create a VFS 202.The user interface may request and receive deployment parameters fromthe administrator, such as the number of FSVMs to be included in the VFS202. In particular embodiments, the deployment request may be receivedby the deployment system. The deployment request may include thedeployment parameters provided by the administrator, such as a name forthe VFS 202, how many FSVMs 170 to include in the VFS 202, networkdetails, such as network addresses for internal and external networks,and name service details, such as an ACTIVE DIRECTORY name, user name,and credentials. Other deployment parameters may be determinedautomatically, e.g., from the cluster, without being requested from theuser. Such automatically-determined parameters may include network timeprotocol information and other configuration details.

In particular embodiments, the deployment image may be provided to eachhost machine 201 via the network from a source such as a deploymentserver. The deployment image may be provided to each host machine 201 bya snapshot operation that creates a snapshot of the deployment image foreach host machine 201. The snapshot operation may provide the datacontained in the deployment image to other host machines 202 using acopy-on-write technique in which the host machines 202 are provided withread access to the image without copying the image. Subsequently, if anyof the host machines 202 were to write to the image (which is notordinarily permitted for deployment images), then the written data maybe stored in the form of changes (e.g., deltas) from the image. Snapshotoperations are described in U.S. Pat. No. 9,009,106, which is herebyincorporated by reference in its entirety. As a result of the snapshotoperation, the deployment image appears, at least to the user VMs (UVMs)105 on the host machines 201, to be locally accessible from the localstorage 122 of the host machine 201. The snapshot operation may make thesnapshot of the deployment image accessible to each host machine withoutany detectable delay.

The deployment image may be used to create a parent snapshot thatincludes the contents of the deployment image. Instead of copying theentire deployment image to each host machine, metadata may be sent orcopied to each host machine 201 via the network to create a childsnapshot on the host machine 201. Contents of the parent snapshot may becopied to the local storage of the host machines 202 as needed, whichmay occur after the snapshot has been created. For data that is notcopied from the parent snapshot to a child snapshot, data requests fromthe host machines 202 may be redirected to access the contents of theparent snapshot via network communication.

In particular embodiments, the snapshot operation and copying of theparent snapshot to the local storage of the host machines 202 may beimplemented by the CVMs 110 a-c. The snapshot operation may be performedN times on the parent snapshot, where N is the number of FSVMs 170 to bedeployed, and N FSVMs 170 may be created. The FSVMs 170 may form acluster, which may provide a VFS 202 to other host machines 201. Use ofthe pre-created fileserver image and the snapshot operation, which maybe performed in less than one second in particular embodiments, canreduce the deployment time to be essentially as fast as booting the hostmachines 201. Thus the deployment process may appear, at least to thesystem administrator, to be nearly instantaneous.

In particular embodiments, the deployment system may attempt to placethe FSVMs 170 on host machines 201 that are at different physicallocations, so that two or more host machines are unlikely to fail orbecome unreachable because of an event such as an electrical power ornetwork connectivity interruption. The physical configuration of eachCVM 110 on each host machine may be stored in a configuration databaseand used to determine where to place the FSVMs. For example, if thereare 10 different CVM 110 nodes, and a four-node file server is beingdeployed, four different CVMs 110 on four different host machines atdifferent physical locations may be chosen. If an insufficient number ofhost machines are at different physical locations, then two or moreFSVMs 170 may be located on host machines at the same physical location,e.g., in the same room, building, or city.

In particular embodiments, when the deployment image has been madeavailable to a host machine 201 the deployment system may create a userVM 105 on the host machine 201 and attach the deployment image (e.g.,attach the boot and data vdisks) to the user VM 105. The deploymentsystem may then power-on the user VM 105. The user VM 105 may then loadthe operating system from the deployment image (e.g., from the bootvdisk). One or more IP addresses for the host machine 201 to use may bepassed to the user VM 105. A CVM 110 and an FSVM 170 may be started onthe host machine 201. The CVMs 110 and FSVMs 170 may be monitored. Whenthe CVMs 110 and FSVMs 170 are operational, a cluster may be created.Bootstrap information may be sent to CVMs 110 on other host machines inthe cluster. ACTIVE DIRECTORY credentials may be sent to an ACTIVEDIRECTORY service so that the FSVMs 170 can join an ACTIVE DIRECTORYdomain. Domain Name Service (DNS) entries for the FSVMs are populated inthe ACTIVE DIRECTORY service or other DNS server. Once thoseconfiguration steps are complete, the host machines 202 in the clustermay provide the VFS 202.

FIG. 8 illustrates an example method 800 for deploying a virtualizedfile server. The method 800 may be performed primarily by, for example,a deployment server. The method 800 begins at step 802 by receiving arequest to deploy a virtualized file server on a plurality of hostmachines. The request may be associated with one or more deploymentparameters. Step 804 may select a deployment image for the host machinesbased on the deployment parameters. Step 806 may provide the selecteddeployment image to each host machine via the virtual disk, wherein thevirtual machine controller located on the host machine stores at least aportion of the deployment image on a storage device associated with thehost machine. Step 808 may create a user VM on the host machine. Step810 may attach the deployment image to the user VM. Step 812 may causethe user VM to power-on and boot from the attached deployment image.

Particular embodiments may repeat one or more steps of the method ofFIG. 8, where appropriate. Although this disclosure describes andillustrates particular steps of the method of FIG. 8 as occurring in aparticular order, this disclosure contemplates any suitable steps of themethod of FIG. 8 occurring in any suitable order. Moreover, althoughthis disclosure describes and illustrates an example method fordeploying a virtualized file server including the particular steps ofthe method of FIG. 8, this disclosure contemplates any suitable methodfor deploying a virtualized file server including any suitable steps,which may include all, some, or none of the steps of the method of FIG.8, where appropriate. Furthermore, although this disclosure describesand illustrates particular components, devices, or systems carrying outparticular steps of the method of FIG. 8, this disclosure contemplatesany suitable combination of any suitable components, devices, or systemscarrying out any suitable steps of the method of FIG. 8.

In particular embodiments, computer program code that implements a VFS202 may be upgraded from an existing version to a newer version bypreparing a newer boot disk image containing the newer version of thecomputer program code and creating a snapshot of the newer boot diskimage for each FSVM 170 of the VFS 202. That is, for a VFS 202 thatincludes N FSVMs 170, N snapshots may be created of the boot disk. Toupgrade the VFS 202 to the newer version, the new boot disk may beswapped with the existing boot disk for each FSVM 170. For example, oneach host machine 201, the existing boot disk of each FSVM 170 on thehost machine may be detached from the virtual machine (VM) in which theFSVM 170 executes, and the new boot disk may be attached to that VM. TheFSVM 170 may then be re-booted, e.g., by rebooting the VM. Afterrebooting the FSVM 170, it is running with the newer code from the newboot disk image, and continues serving data using the newer version ofthe computer program code.

FIG. 9 illustrates an example method 900 for upgrading a virtualizedfile server. The method 900 may be performed primarily by, for example,a deployment server. The method 900 begins at step 902 by receiving arequest to upgrade a virtualized file server on a set of host machinesfrom a first program code version to a second program code version. Step904 may, for each host machine, generate a snapshot of a code imageassociated with the second program code version. Step 906 may providethe snapshot to the host machine via the virtual disk. The snapshot maybe provided as a boot disk generated based on the snapshot. Step 908 maydetach an existing code image from a user VM located on the hostmachine. Step 910 may attach the snapshot to the user VM. Step 912 maycause the user VM to boot from the snapshot.

In particular embodiments, computer program code that implements a VFS202 may be upgraded from an existing version to a newer version bypreparing and creating a snapshot of a newer boot disk containing thenewer version of the computer program code, instructing each FSVM 170 aof the VFS 202 to acquire an upgrade token that may be acquired by onlyone FSVM 170 a at a time, and, when an FSVM 170 a acquires the token,swapping the old boot disk with the newer boot disk on the FSVM 170 a,rebooting the FSVM 170 a as described above, and relinquishing theupgrade token. That is, when the first FSVM 170 a comes back up and isrunning, the upgrade token is passed to the next FSVM 170 b, which mayperform the swap and reboot, and pass the upgrade token to the next FSVM170 c. These operations are repeated until the last FSVM 170, e.g., FSVM170 c in this example, is upgraded. During the time that each FSVM 170 bis being rebooted, one of the peer FSVMs 170, e.g., FSVM 170 a may takeover the storage and IP address of the FSVM 170 b, so that the clientsystem does not detect any interruption in the file service provided bythe VFS 202.

FIG. 10 illustrates an example method 1000 for performing a rollingupgrade of a virtualized file server. The method 1000 may be performedprimarily by, for example, a deployment server. The method 1000 beginsat step 1002 by receiving a request to upgrade a virtualized file serveron a plurality of host machines from a first program code version to asecond program code version. Step 1004 may, for each host machine,request an upgrade token. Step 1006 may, when the host machine acquiresthe upgrade token, generate a snapshot of a code image associated withthe second program code version. Step 1008 may provide the snapshot tothe host machine via the virtual disk. The snapshot may be provided as aboot disk generated based on the snapshot. Step 1010 may detach anexisting code image from a user VM located on the host machine. Step1012 may attach the snapshot to the user VM. The user VM may boot fromthe snapshot. Step 1014 may release the upgrade token acquired by thehost machine.

FIG. 12 illustrates example virtualized file server operations 1200,including data migration operations. In particular embodiments, datamigration may involve transferring existing files, which may beorganized in hierarchies of directories accessible via network shares1204, or the like, from an existing virtual file server (VFS) 1202 to anew VFS 1208 via a network 140. Since the existing and new VFSs may beunavailable while the migration is in progress, the migration should becompleted quickly to minimize system downtime. However, data migrationmay consume a substantial amount of time, particularly for large amountsof data. The performance of a data migration system may be restricted bythe speed of network communication between existing infrastructure(e.g., host machines 1201 of the existing VFS 1202) and the new system(e.g., host machines 1207 of the new VFS 1208). Data migration may beperformed when, for example, an existing VFS 1202 is to be replaced by anew VFS 1208, or existing host machines 1201 are to be replaced by newhost machines 1207. By using the data migration techniques disclosedherein, migration speed can be increased by a multiplier of the numberof file server host machines 1201.

In previous approaches, data may be migrated from an existing, e.g.,source, File Server Virtual Machine (FSVM) 1203 a in an existing, e.g.,source, VFS 1202 using a utility to copy data from one source to onetarget location. Migration speed is limited by the connection speed. Inparticular embodiments, using the smart data ingestion approachdescribed herein, directories in an existing VFS 1202 are processed toacquire the new, e.g., destination, FSVM 1209 a in a new, e.g.,destination, VFS 1208. Note that the terms “existing” and “new” are usedherein for explanatory purposes to distinguish between differententities, such as virtual file servers or FSVMs and do not necessarilyrelate to age of the entities being referred to. In particularembodiments, the new VFS 1208 has a set of new FSVMs 1209 that maycorrespond to the existing FSVMs 1203 of the existing VFS 1202. Althoughone FSVM 1209 a is shown in the new VFS 1208 in FIG. 12, the new VFS1208 may contain multiple FSVMs. The new FSVMs 1209 may be created aspart of the splitting operation or may be existing FSVMs that werepreviously created. For example, the new VFS 1202 may have the samenumber of FSVMs 1209 as the existing VFS 1202, and each FSVM 1209 in thenew VFS 1208 may correspond to one of the existing FSVMs 1203 in theexisting VFS 1202. In other examples, the existing and new VFSs may havedifferent numbers of FSVMs, but a mapping may still exist from existingFSVMs 1203 to new FSVMs 1209, and the mapping may be used to identifythe new FSVM 1209 to which each data object is to be migrated. That is,the mapping that establishes a correspondence between existing FSVMs1203 and new FSVMs 1209 need not be one-to-one.

In particular embodiments, when data migration begins, each existingFSVM 1203 of the existing VFS 1202 may start migration of shares 1204assigned to the existing FSVMs 1203. The example shares 1204 a assignedto FSVM 1203 a include an \Office\Admin\HR share and an\Office\Admin\Finance share. The shares 1204 b assigned to FSVM 1203 binclude a \Development share. Each share 1204 may correspond to adirectory that stores storage items such as files and other directories.In particular embodiments, migration may be performed to make data fromone or more existing FSVMs 1203 in the existing VFS 1202 available tocorresponding “new” FSVMs 1209 in the new VFS 1208. The new FSVM 1209 amay be located on a corresponding host machine 1207 a in the new VFS1208. The mapping of new FSVMs 1209 to available host machines 1207 maybe performed by a load-balancing component, system admiration interfacebased on a system administrator's input, or other suitable component.The host machine 1207 a in the new VFS 1208 that corresponds to theexisting host machine 1201 a may be identified using a mapping from thehost machines 1201 of the existing VFS 1202 to the host machines 1207 ofthe new VFS 1208. The mapping may be, e.g., a lookup table or a namingor addressing convention. That is, the corresponding host machines 1207in the new VFS 1208 may have host names or addresses derived from thehost names or addresses of the machines in the existing VFS 1202. Thus,a host machine 1201 a named “host1-vfs1” in the existing VFS 1202 maycorrespond to a host machine 1207 a named “host1-vfs2” in the new VFS1208.

As another example, corresponding hosts 1201 a, 1207 a in different VFSs1202, 1208 may have IP addresses in different subnets, e.g., a host withthe address 128.1.1.1 in an existing VFS 1202 subnet 129.1.1.X maycorrespond to a host with the address 128.1.2.1 in a new VFS 1208 subnet128.1.2.X, where X has the same value for corresponding host machines1201 a and 1207 a. In another example, a sharding algorithm may havebeen used to determine the FSVMs 1203 or host machines 1201 thatcorrespond to each share 1204 (or other storage item, e.g., directory orfile) for the existing VFS 1202. The same sharding algorithm may be usedto determine the FSVMs 1209 or host machines 1207 that correspond toeach share 1210 for the new VFS 1208.

Because of the distributed nature of the VFS 1202, data migration may beperformed in parallel to speed up the migration process. In particularembodiments, multiple data objects, e.g., Shares 1204, may be migratedin parallel (e.g., concurrently with each other), thus speeding up datamigration. Data migration may be sped up by a multiple of the FSVMcount. For example, if there are four FSVMs in the existing VFS and fourFSVMs in the new VFS, then the data migration may be sped up by a factorof four because four transfers may proceed in parallel, e.g., if adedicated processor is available to execute the operations for each ofthe eight FSVMs. Transfers between each pair of FSVMs (e.g., theexisting FSVM and the corresponding new FSVM) may be performed inparallel with transfers between other pairs of FSVMs (and there are fourpairs of FSVMs in this example). In another example, individual storageitems, such as files and directories in shares 1204, may have associatedfile names and paths (e.g., directory paths) in the existing VFS 1202,and may be transferred to the new FSVMs and stored with the same filenames and same paths on the new FSVMs. Alternatively, the paths on thenew FSVMs may be based on but different from the paths on the existingFSVMs. For example, a file on the existing FSVM 1203 a named Report.docin the directory \Office\HR of a share 1204 a may be transferred to thenew FSVM 1209 a and stored in the directory \ HR of a share 1210 a onthe new FSVM 1209 a or in a different directory on the new FSVM 1209 a,e.g., \HumanResources. The file name may remain the same on the new FSVM1209 a, e.g., Report.doc.

As introduced above, the term “storage item” may be used herein to referto a share 1204, a file, or a directory. Alternatively, a storage itemmay be a different unit of data, such as a disk block. The placement ofdata, e.g., the mapping of files, directories, blocks, or other storageitems, to the locations at which the data are to be placed, e.g., FSVMs,for file I/O, may be determined by a sharding algorithm. The shardingalgorithm may generate a sharding map 360 a by, for example, computing ahash value based on the data to be placed and generating an FSVMidentifier or address based on the hash value (e.g., by computing thehash value modulo the number of FSVMs in the VFS 1202). As anotherexample, the sharding algorithm may generate a sharding map 360 a basedon the capacity of the FSVMs (or the capacity of the host machines onwhich the FSVMs execute), e.g., based on the amount of disk spaceavailable, the number and processing speed of CPUs of host machines1201, 1207, the amount of memory, or other measures of capacity. Thesharding algorithm may determine an amount of capacity needed orpreferred for each storage item, e.g., the size of the directory or filerepresented by the storage item, or a value based on a historicalmaximum size of the directory or file, or the like. The shardingalgorithm may then determine a corresponding FSVM for each storage itemby, for example, allocating storage items to FSVMs that have sufficientavailable capacity for the storage items, then reducing the availablecapacity by the size (or maximum size, or other measure) of the storageitem. A brute-force resource allocation technique may be used, e.g., byallocating storage items to FSVMs in various combinations until asatisfactory allocation in which the storage items fit within thestorage capacity of the FSVMs is found.

In particular embodiments, the sharding algorithm may determine alocation for each storage item in the existing VFS 1202. The locationmay be, e.g., a particular FSVM 1203, which may correspond to aparticular host system 1201. Each storage item may be, e.g., a directoryor file in the existing VFS 1202. A list of the determined locations,e.g., FSVMs 1203, may be stored by the existing VFS 1202, e.g., when thelocations are determined. When migration is performed, the list ofdetermined locations may be retrieved from the existing VFS 1202 or maybe generated using the sharding algorithm based on the existing VFS1202. The list of determined locations may be represented as a shardingmap 360 a generated by the sharding algorithm.

An example sharding map that includes locations of two storage items,File-1 and File-2, in an existing VFS 1202 named fs0, is shown in Table2 below. The location of existing File-1 is FSVM1 1203 a, and thelocation of existing File-2 is FSVM2 1203 b. Although two files areshown in Table 2, any number of files and/or directories and theircorresponding locations may be included in the sharding map.

TABLE 2 Existing VFS fs0 Storage item Location File-1 FSVM1 File-2 FSVM2

In particular embodiments, once an existing storage item's location isknown, the storage item may be transferred from its location in theexisting VFS 1202 to the corresponding location in the new VFS 1208without waiting for other storage items to be transferred. Thecorresponding location in the new VFS 1208 may be determined by, forexample, using a lookup table that specifies the locations of storageitems in the new VFS 1208. As another example, the correspondinglocation in the new VFS 1208 may be determined by transforming the name,address, or identifier of the location in the existing VFS 1202 asdescribed above. The transformation may be done by, e.g., by changingthe virtual file server name fs0 to fs1 in names of host machines, orchanging the subnet value in IP addresses of host machines, e.g., from 1to 2.

In particular embodiments, a list of the determined locations may bestored by the existing VFS 1202, e.g., when the locations aredetermined. When migration is performed, the list of determinedlocations may be retrieved from the existing VFS 1202 or may begenerated using the sharding algorithm based on the existing VFS 1202.Alternatively, the locations of the storage items in the new VFS 1208may be determined using the same sharding algorithm that was used todetermine the locations of the corresponding storage items in theexisting VFS 1202. For example, a sharding map 360 that includes thelocations of the example storage items File-1 and File-2 in the new VFS1208 (named fs1) are shown in Table 3 below.

TABLE 3 New VFS fs1 Storage item Location File-1 FSVM11 File-2 FSVM11

In the example introduced above, the location of new File-1 is fs1'sFSVM11 1209 a (which corresponds to fs0's FSVM1 1203 a), and thelocation of new File-2 is also fs1's FSVM1 1 (which corresponds to fs0'sFSVM2 1203 b).

In particular embodiments, transfers of two or more storage items mayproceed in parallel. The existing VFS 1202 may initiate and perform upto N of the transfers in parallel, wherein N is a configurable limitbased on the processing and storage capacities of the host machines onwhich the FSVMs 1203 of the VFS 1202 execute. For example, multiple hostmachines may access their corresponding local storage devices inparallel. A transfer of File-1 from FSVM1 1203 a on host machine 1201 ato FSVM11 on host machine 1207 a may be performed in parallel with atransfer of File-2 from FSVM2 on host machine 1201 b to FSVM1 1 on hostmachine 1207 a. The data (e.g., blocks) of File-1 may be read from hostmachine 1201 a's local storage, sent to host machine 1207 a via thenetwork, and written to host machine 1207 a's local storage while, inparallel, the data of File-2 is read from host machine 1201 b's localstorage, sent to host machine 1207 a via the network, and written tohost machine 1207 a's local storage.

FIG. 11 illustrates an example method 1100 for ingesting data into avirtualized file server 1208. The method 1100 may be performed primarilyby, for example, one of the host machines 201. The method 1100 begins atstep 1102 by connecting to an existing virtualized file server (VFS)1202. Step 1104 may retrieve a list of existing storage items 1204,e.g., shares, directories, and/or files, that are located on theexisting VFS 1202, and their sizes. The list may be retrievedrecursively to identify nested directories. Step 1106 may identify thelocations on the existing VFS 1202 of the existing storage items. Thelocations may be, e.g., the addresses or identifiers of the FSVMs 1203on which the storage items are located, and may also include path namesof the files and directories that are stored in nested directories. Thelocations may be retrieved from a sharding map 360 a associated with theexisting VFS 1202. Step 1108 may generate a list of locations at whichthe storage items 1204 are to be stored in a new VFS 1208. The list oflocations may be based on the existing locations (e.g., identities ofFSVMs 1203 and path names) of the existing storage items 1204. Forexample, the locations of the existing storage items identified at step1106 may be fed to the sharding algorithm to determine the particularFSVM 1209 of the new VFS 1208 on which each file and/or directory is tobe stored. The sharding algorithm may generate a sharding map 360 forthe new VFS 1208 based on the identified locations. Alternatively, thelocation at which each new file and/or directory is to be stored on thenew VFS 1208 may be determined according to a naming or addressingconvention, or other mapping, and the determined locations may be storedin a sharding map 360 a. That is, the sharding map 360 a need not begenerated directly by a sharding algorithm.

At step 1110, the data, e.g., the storage items, may be migrated to thenew VFS 1202 by transferring each existing storage item from itslocation (e.g., existing FSVM 1203 and path) in the existing VFS 1202 toits location in the new VFS 1208. The transfer may be implemented bysending the storage item and associated metadata, such as the storageitem's name and permissions, from the existing FSVM 1203 to the new FSVM1209 at which the new storage item is to be located. Each new file ordirectory may be created on the new FSVM 1209 in the new VFS 1208 usingthe same file name or directory name that was used in the existing VFS202, or using a different file name or directory name if specified. Thetransfers of files between different pairs of FSVMs may be performed inparallel. For example, the transfers may be requested in parallel(without waiting for a transfer to complete prior to starting anothertransfer), and the FSVMs 1203 or other components may determine whichtransfers can be performed independently of each other, and execute theindependent transfers in parallel. After the transfers are complete, thedata, e.g., the storage items, are ready to be served by the new VFS1208.

FIG. 12 illustrates example virtualized file server (VFS) operations1200, including VFS splitting and merging operations. In particularembodiments, an existing virtualized file server (VFS) 1202 may be splitinto multiple new virtualized file servers (VFSs) 1208, 1212, 1216. Asan example, an enterprise or organization may have an existing VFS 1202that serves storage items such as shares, directories, or files toclients in several departments of the organization. The organization maydecide to split the existing VFS 1202 into multiple new VFSs 1208, 1212,1216, e.g., to move the file-serving workload of particular departmentsto host machines managed by or otherwise associated with thosedepartments. As an organization grows, the number of users of theexisting VFS 1202 may increase, thereby increasing the workload of theVFS 1202. By splitting the VFS 1202 into multiple new VFSs, the workloadmay be distributed across the new VFSs. Assigning the new VFSs todepartments in an organization may simplify administration andmanagement of the file servers. Conversely, multiple VFSs 1208, 1212,1216 may be merged together to form a single merged VFS 1208. Forexample, when an organization that manages multiple VFSs for differentdepartments decides to consolidate the VFSs into one VFS 202, a mergingoperation may be performed to merge the multiple VFSs together

Previously, when an organization decided to split a file server, asystem administrator deployed new file servers and migrated the datafrom the old file server to the newly deployed file servers. Conversely,when an organization decided to merge multiple file servers, the systemadministrator deployed a new file server and migrated stored data fromthe multiple file servers to the newly deployed file server. Suchexisting techniques involve deployment and migration processes that mayuse substantial amounts of time, storage, and network resources foroperations such as copying data between storage devices or betweenservers across a network. Thus substantial file server downtime mayoccur as a result of delays caused by the existing and new file serversbeing unavailable while split or merge operations are in progress.

In particular embodiments, virtualized file servers may be split ormerged while incurring substantially less downtime than in existingtypes of file servers. For example, splitting or merging VFSs may beperformed efficiently by using a snapshot operation to provide the datalocated at each existing File Server Virtual Machine (FSVM) of theexisting VFS to one or more new FSVMs of one of the new VFS withoutcopying the data. The new FSVMs 1209 may be FSVMs that have previouslybeen created or that are newly created, e.g., as part of the splittingprocess, for use in the new VFSs 1208, 1212, 1216. In particularembodiments, each new VFS created from an existing VFS 1202 may includeone or more new or pre-existing FSVMs 1209 that are configured based onconfigurations of corresponding existing FSVMs 1203 of the existing VFS202. The new VFSs 1208, 1212, 1216 do not ordinarily include any of theexisting FSVMs 1203 from the existing VFS 1202. In particularembodiments, the existing VFS 1202 may continue to operate after thesplitting operation, e.g., by retaining one or more of its FSVMs 1203and continuing to serve one or more shares 1204. Although three new VFSs1208, 1212, 1216 are shown in the example of FIG. 12, an existing VFS1202 may be split into any suitable number of new VFSs.

In particular embodiments, when the splitting operation is performed,the FSVMs 1209 to be included in each new VFS 1208, 1212, 1216 may beselected by a system administrator, by a user, or by an automaticprocess. For example, a system administrator may specify how many newVFSs are to be created, and how many FSVMs 1209 are to be included ineach of the new VFSs. Thus, each FSVM 1209 may be incorporated into oneof the new VFSs according to an FSVM-to-VFS mapping. Further, thestorage items 1210 served by each new VFS 1208, 1212, 1216 may be thestorage items 1204 that were being served by the FSVMs 1203 of theexisting VFS 1202. Optionally, the system administrator may selectparticular storage items to be served by the new VFSs 1208, 1212, 1216.The host machines on which a new VFS is located may be specified, e.g.,by a system administrator or by a load-balancing system. The hostmachines on which the new VFS is located may be physically located at ornear the physical location of the host machines used by the existing VFS202, or may be at other physical locations, e.g., at or near thephysical location of a department to which a new VFS is assigned.

In particular embodiments, a VFS splitting operation may be performed tosplit an existing VFS 1202 into multiple new VFSs 1208, 1212, 1216 by,for each new VFS to be constructed, selecting one or more existing FSVMs1203 to be removed from the existing VFS 1202, and, for each existingFSVM, identifying one or more existing or (newly-created) available,e.g., lightly-loaded, FSVM to be included in the new VFS, re-assigningresources from the existing FSVM to one or more other available FSVMs1209, and re-directing subsequent client requests for the existing FSVM1203 to the available FSVMs 1209. The existing available FSVMs 1209 maybe previously-created FSVMs 1209 that are lightly loaded, or may becreated, e.g., if there are no existing available FSVMs. If an FSVM 1209is created, it may be created on a lightly loaded host machine 1207. Thesplitting operation may further include removing the existing FSVM 1203from the existing VFS 1202, moving storage items that were notre-assigned, e.g., shares 1204, files, folders, or the like, beingaccessed by each existing client of the existing FSVM 1203 from theexisting FSVM 1203 to the available FSVM 1209, and, for those movedstorage items, re-directing or re-connecting clients of the existing VFS1202 to the available FSVM 1209. The re-assigned resources may includecompute resources (e.g., those used by the FSVM), storage resources(e.g., disk space allocated to the FSVM), and metadata resources (e.g.,ACTIVE DIRECTORY credentials). Resources may be assigned from theexisting FSVMs 1203 to the available FSVMs 1209 automatically, e.g., byassigning each resource from the existing FSVM 1203 to a correspondingavailable FSVM 1209, or by assigning resources to available FSVMs 1209according to a mapping provided by a system administrator or generatedby a load-balancing system.

In particular embodiments, splitting a VFS may be performed efficientlywhen the existing VFS 1202 and the new VFSs 1208, 1212, 1216 useexisting FSVMs 1203 and available FSVMs 1209 that are in the samecluster, in which case data need not be copied between the existing andavailable FSVMs. Otherwise, if one or more of the existing FSVMs 1203are not in the same cluster as one or more of the available FSVMs 1209,then data may be copied between those FSVMs 1203 and FSVMs 1209 that arein different clusters. The data may be copied from storage associatedwith the existing FSVMs 1203 to storage associated with the availableFSVMs 1209. Thus, for example, shares may be moved between VFSs withzero data copying or migration by re-assigning the data to availableFSVMs 1209 in the new VFS if the existing FSVMs 1203 and available FSVMs1209 are in the same cluster. Users may perceive little or no down timeof shares 1204 re-assigned from the existing VFS to the new VFS. Thedata that may be copied can include storage items such as files orfolders associated with the existing FSVMs 1203, e.g., files or foldersstored on shares served by the existing FSVMs. For example, the existingVFS 1202 may serve one or more network-shared filesystems, e.g., SMBshares, NFS exports, or the like. The network shared filesystems orshares served by the existing VFS 1202 may be re-assigned from theexisting VFS 1202 to the new VFS 1208 without copying data on thefilesystems if the existing FSVM 1203 hosting the filesystem is in thesame cluster as the available FSVM 1209 in the new VFS 1208 that is tohost the filesystem.

In particular embodiments, an available FSVM 1209 may be a backup FSVMthat provides a high-availability backup for an existing FSVM 1203.Resources of the existing FSVM 1203, such as storage resources, may beavailable on the backup FSVM, e.g., because the resources on the backupFSVM, such as data on the storage device, are kept up-to-date with theexisting FSVM 1203. Resources of the existing FSVM 1203 may bere-assigned to the corresponding backup FSVM by performing a failoverfrom the existing FSVM 1203 to the backup FSVM. The backup FSVM 1203 maythen become the available FSVM 1209 in the new VFS 1208. Since thebackup FSVM maintains an up-to-date copy of the data hosted by theexisting FSVM 1203, the backup FSVM may be used in the new VFS 1208 withresources of the existing FSVM 1203 without copying the resources, e.g.,data that represents the resources, from the existing FSVM 1203 to thebackup FSVM as part of the splitting operation.

In particular embodiments, network file access requests directed to anexisting FSVM 1203 may be re-directed to the available FSVM 1209 thatreplaces the existing FSVM 1203. For example, the IP address of anexisting FSVM may be moved to the corresponding available FSVM, e.g.,assigned to the available FSVM to maintain client connections such asSMB connections. When an FSVM 1203 is removed from the existing VFS1202, the system may wait until the backup FSVM that corresponds to theremoved FSVM 1203 has become a primary FSVM, and is ready to performfile serving operations as an available FSVM 1209, before re-directingrequests from clients to the backup FSVM and before removing the nextexisting FSVM 1203 in the sequence.

In particular embodiments, clients accessing storage items served by theexisting VFS 1202 may continue performing access operations, e.g., readsand writes, on storage items that were being served by the removed FSVM1203 prior to its removal. The clients of the existing VFS 1202 maycontinue to access storage items if the storage items are moved to anavailable FSVM 1209 in the new VFS 1208 if, for example, the clients arere-directed to the available FSVM 1209 by moving the existing FSVM's IPaddress to the available FSVM or by DFS redirections. Alternatively,e.g., if a client has an existing connection to an existing FSVM 1203that cannot be re-directed, e.g., the client is accessing shares using acommunication connection that cannot be redirected, then the client maybe requested to close the existing FSVM connection, e.g., close theshares and re-map the shares to the client machine. Thus the existingVFS 1202 may be split without affecting access to the stored data, atleast until a later time at which existing client connections to thestorage items are closed, e.g., in response to a request from a systemadministrator.

As an example, there may be an existing VFS 1202, named fs0, which ishosted by nine FSVMs 1203 named FSVM1-FSVM9. These FSVMs 1203 may belocated on (e.g., executing on) two existing host machines 1201 a,b in acluster. Thus, fs0 has nine existing FSVMs as follows:

-   -   fs0=[FSVM1, FSVM2, FSVM3, FSVM4, FSVM5, FSVM6, FSVM7, FSVM8,        FSVM9]

Three shares 1204, named HR, Finance, and Development, are provided byfs0 in this example. The HR and Finance shares are served by FSVM1 1203a, and the Development share is served by FSVM2 1203 b. These exampleshares are shown in Table 4.

TABLE 4 Share Path VFS FSVM \\fs0\Office\Admin\HR fs0 FSVM1\\fs0\Office\Admin\Finance fs0 FSVM1 \\fs0\Development fs0 FSVM2

A system administrator may request that the resources of the nine FSVMs1203 initially allocated to the fs0 VFS 1202 be divided among three newvirtualized file servers 1208, 1212, and 1216, named fs1, fs2, and fs3,respectively. That is, the nine FSVMs 1203 are removed from fs0, but arenot added to the new VFSs 1208, 1212, 1216. Instead, the resources ofthe nine FSVMs 1203, such as shares 1204, disk storage, credentials, andso on, are provided to available FSVMs 1209 that are not part of fs0,and the available FSVMs 1209 are used by the new VFSs 1208, 1212, 1216.The particular available FSVMs 1209 that are assigned to the new VFSs1208, 1212, 1216 may be specified by the system administrator or may beselected automatically (e.g., divided as evenly as possible among thenew VFSs 1208, 1212, 1216). The host machines 1207 to be used for thenew VFSs may be identified implicitly as the host machines on which theselected FSVMs are located. Alternatively, the administrator may specifyhost machines 1207 on which the new VFSs are to be located, andavailable FSVMs 1209 on the specified host machines 1207 may be used forthe new VFSs. If there are too few available FSVMs 1209, then additionalFSVMs 1209 may be created, e.g., on specified host machines 1207 or onsuitable available host machines 1207 if none are specified. Further,the administrator may specify a mapping between the existing FSVMs 1203and the new VFSs 1208, 1212, 1216. For example, each of the three newVFSs may be assigned three available FSVMs 1209, which may subsequentlybe associated with corresponding existing FSVMs 1203. The resources ofFSVM1-FSVM9 may then be assigned to available FSVMs FSVM11-19,respectively. FSVM11-19 may be assigned to the new VFSs according to amapping specified by the administrator, or may be distributedautomatically, e.g., as evenly as possible across the new VFSs. In thisexample, FSVM11-FSVM13 are assigned to fs1, FSVM14-FSVM16 are assignedto fs2, and FSVM17-FSVM19 are assigned to fs3. This mapping of availableFSVMs 1209 to new VFSs 1208, 1212, 1216 is shown below.

-   -   fs1=[FSVM11, FSVM12, FSVM13]    -   fs2=[FSVM14, FSVM15, FSVM16]    -   fs3=[FSVM17, FSVM18, FSVM19]

The virtualized file server splitting operation may split the existingVFS 1202 (named fs0) into the new VFSs 1208, 1212, 1216 (named fs1, fs2,fs3, respectively) without causing interruptions in service to clientsof the existing VFS 1202.

In the example described above, the new VFSs 1208, 1212, 1216 arecreated by splitting the existing VFS 1202. There are three availablehost machines 1207 a-c, and the nine available FSVMs 1209, namedFSVM11-FSVM19, are located on the three available host machines 1207a-c. The existing FSVMs 1203, named FSVM1-FSVM9, are ordinarily notincluded in any of the new VFSs 1208, 1212, 1216. Instead, availableFSVMs 1209 are used in the new VFSs and configured based on theconfigurations of the existing FSVMs 1203. Each of the storage items,e.g., shares 1204, associated with each existing FSVM 1203, such as theHR, Finance, and Department shares, may be provided to a correspondingone of the available FSVMs as shares 1210. The stored data referenced bythe shares 1204 a, including \Office\Admin\HR on FSVM1 (e.g., the filesand folders in the HR directory) may be made accessible to FSVMs 1209 inthe new VFSs without copying if, e.g., FSVM1 is in the same cluster asFSVM1 1 (to which the \HR share is moved and appears as share 1210 a)and FSVM12 (to which the finance share is moved and appears as share1210 b). Alternatively, if FSVM1 is not in the same cluster as FSVM1 1and FSVM12, then data referenced by the shares 1203 a may be madeaccessible to FSVM1 1 and FSVM12 by being copied from storage, e.g.,storage pool 160, associated with FSVM1 to storage associated with FSVM11 and FSVM12, e.g., a storage pool in a different cluster, such as localstorage 122 of the host machines 1207 that are in a different clusterthan the host machines 1201.

In particular embodiments, according to an example mapping fromavailable FSVMs 1209 to available host machines 1207, which may beprovided by an administrator or automatically generated, three of theavailable FSVMs 1209 may be located on each of the available hostmachines 1207. Although the example VFS 1202 is split into three newVFSs 1208, 1212, 1216 located on three different host machines 1207 a,1207 b, 1207 c in this example, a VFS 1202 may be split into any desirednumber of new VFSs located on any desired number of host machines 1207.Further, although a one-to-one mapping of existing FSVMs 1203 toavailable FSVMs 1209 is described in examples herein, any mapping ofexisting FSVMs 1203 to available FSVMs 1209 may be used, e.g., threeexisting FSVMs 1203 in an existing VFS 1202 serving a total of 100shares may be mapped to four new VFSs, each having 2 host machines 1207with 1 FSVM 1209 located on each host machine. The 100 shares may bedivided into 25 shares for each of the four new VFSs, and the 25 sharesof each new VFS may be divided into, e.g., 10 shares on the first hostmachine and 15 shares on the second host machine of the new VFS. Eachavailable FSVM 1209 may use storage of the storage pool 160, e.g., localstorage 122 provided by the host machine 1207 on which the FSVM 170 islocated, cloud storage 126, or networked storage 128.

In particular embodiments, an existing VFS 1202 may initially belong toor be used by one or more departments or other entities in anorganization or enterprise, and may be serving a substantial number(e.g., hundreds) of shares accessed by client systems. The existing VFS1202 may be used to store data for several administrative departments ina corporation, for example. A system administrator may decide to splitthe VFS 1202 into multiple VFSs, each of which is to be assigned to oneof the departments. For example, the existing VFS 1202 (named fs0) maybe split into three new virtualized file servers: a VFS 1208 (named fs1)for the HR department, a VFS 1212 (named fs2) for the Financedepartment, and a VFS 1216 (named fs3) for the Development department.As a result of the example splitting operation, the example shares shownon the fs0 virtualized file server in TABLE 1 above (HR, Finance, andDevelopment) may be moved to the three new virtualized file servers asshown in Table 5.

TABLE 5 Share Path VFS FSVM \\fs1\HR fs1 FSVM11 \\fs2\Finance fs2 FSVM12\\fs3\Development fs3 FSVM13

The HR share has been moved to the VFS 1208 named fs1 on FSVM11, theFinance share has been moved to the VFS 1212 named fs2 on FSVM12, andthe Development share has been moved to the VFS 1216 named fs3 on FSVM13.

In particular embodiments, each of the existing FSVMs 1203 (namedFSVM1-FSVM9) may be removed from the existing VFS 1202, and theresources of each existing FSVM 1203 may be moved to an available FSVM1209 that is not included in the existing VFS 1202. The existing FSVMs1203 may be removed sequentially, e.g., one-by-one, in a sequence havingany appropriate order. For example, the existing FSVMs 1203 may beremoved in the order in which they are selected by the administrator, orin order of their numeric identifiers (e.g., FSVM1 may be removed first,followed by FSVM2, and so on).

To split the existing VFS 1202 (named fs0) into the three new VFSs 1208,1212, and 1216 (named fs1, fs2, and fs3, respectively), FSVM1-FSVM3 maybe removed from fs0 and added to the new fs1. Available FSVMs 1209,named FSVM11, FSVM12, and FSVM13, may be used or created on the hostmachines 1207. Each of FSVM11, FSVM12, and FSVM13 may be located on arespective one of the host machines 1207 a, 1207 b, and 1207 a.Alternatively FSVM11, FSVM12, FSVM13 may be located on a single hostmachine 1207 a, on two host machines 1207 a, 1207 b, or on any othersuitable number of host machines. Similarly, FSVM4-FSVM6 may be removedfrom fs0 and added to the new fs2 as FSVM14, FSVM15, and FSVM16 on hostmachine 1207 b, and FSVM7-FSVM9 may be removed from fs0 and added to thenew fs3 as FSVM17, FSVM18, and FSVM19 on host machine 1207 c.

In particular embodiments, compute units (e.g., an FSVM), networking(e.g., IP addresses) and storage associated with the FSVM may be movedfrom an existing host machine 1201 to a new host machine 1207. Moving anexisting FSVM 1203 may involve creating a new FSVM 1209 on an availablehost machine 1207 based on the existing FSVM 1203, e.g., based on theconfiguration of the existing FSVM 1203. Creating the new FSVM 1209 mayinvolve moving the IP address of the existing host machine 1201 to theavailable host machine 1207, and relinquishing storage resourcesallocated to the existing FSVM 1203 by the existing host machine so thatthe resources may be used by one or more available FSVMs 1209 on theavailable host machine 1207. Storage items used by the existing FSVM1203 may be stored on storage devices that can include one or more ofthe cloud storage 126, the networked storage 128, or the local storage122. The available FSVM 1209 may access these storage resources. Storageitems may be moved to another storage device, if appropriate, while theexisting FSVM 1203 continues to access the storage items at theirexisting locations, even after the new FSVM 1209 has been created on theavailable host machine 1207. The existing FSVMs 1203 may continue toaccess the storage items at their initial locations until the suchaccess is disabled, e.g., by a user closing the shares 1204 that areaccessing the storage items, which the user may do in response to arequest from a system administrator, or by the system administratorinvoking a command to close those shares. When the existing FSVM 1203can no longer access the storage items, e.g., because the shares 1204have been closed or the storage items have otherwise becomeinaccessible, the storage items may be accessed via the new FSVM 1209 attheir new locations, e.g., as shares 1210. If the storage items arestored on local storage 122 of the existing host machine, then movingthe storage items to the local storage 122 of the new host machine mayresult in improved performance of the FSVM 1209 on the new host machine1207. In particular embodiments, ACTIVE DIRECTORY credentials, e.g., foruse by DFS, may be moved from the existing host machine 1201 to the newhost machine 1207 for access by the available FSVM 1209.

FIG. 13 illustrates an example method 1300 for splitting an existingvirtualized file server (VFS) 1202 into two or more new virtualized fileservers (VFSs) 1208, 1212, 1216. The method 1300 may be performedprimarily by, for example, a deployment server, a host machine 1207 of avirtualized file server, or other component associated with avirtualized file server 1202. The method 1300 may be used to split anexisting VFS 1202 in response to a system administrator's request, forexample. The method 1300 begins at step 1302 by selecting one or moreexisting FSVMs 1203 to be removed from the existing VFS 1202. Eachselected existing FSVM 1203 is to be added to one of the new VFSs. Foreach of the existing FSVMs 1203, step 1304 may identify an availableFSVM 109, such as a lightly-loaded FSVM 1209, which is separate from theexisting VFS 1202. The available FSVM 1209 may be, e.g., an FSVM locatedon a host machine 1207 that has an average or current CPU utilizationrate below a threshold and/or is serving less than a threshold number ofshares 1210, and is not one of the existing FSVMs 1203. The CPUutilization rate may be the percentage of time or CPU cycles duringwhich the CPU is executing application instructions or is not idle, forexample. The threshold CPU utilization may be, e.g., 5%, 10%, 20%, orthe like. The threshold number of shares may be, e.g., 1, 5, 10, 25, 50,100, or the like. As another example, the available FSVM 1209 may be abackup FSVM of the existing FSVM 1203. The backup FSVM may have accessto resources of the existing FSVM 1203, and may be capable of accessingthe resources independently of the existing FSVM 1203.

Step 1306 may provide one or more resources of the selected FSVM 1203 tothe available FSVM 1209. For example, the selected FSVM's storageresources may be located on one or more of the local storage 122, cloudstorage 126, and networked storage 128. The storage resources may beprovided to the available FSVM 1209 by, for example, changing allocationinformation so that the storage resources are associated with theavailable FSVM 1209. The resources may include storage resources, e.g.,storage capacity such as that provided by the storage pool 160.Providing the resources may include providing data stored on the storageresources to the available FSVM 1209. The data may be provided withoutbeing copied, e.g., by providing access via the network between theexisting FSVM 1203 and the available FSVM 1209 if the existing andavailable FSVMs are located in the same cluster, or re-assigning thestorage on which the data is stored to the available FSVM 1209. The datamay be provided without being copied at the time the data is provided,e.g., because the data has previously been copied to the available FSVM1209 by high-availability/fault-tolerance features such as thosedescribed elsewhere herein. The data may include one or more storageitems associated or served by with the existing FSVM 1203, and thestorage items may be shares 1203, files, or folders. As another example,the resources may include security credentials, such as ACTIVE DIRECTORYcredentials that permit an FSVM 1209 to join an ACTIVE DIRECTORY domainand participate in DFS referrals.

Step 1308 may re-direct requests to access storage items that are servedby the existing FSVM 1203 to the available FSVM 1209, so that theavailable FSVM 1209 may process and respond to the requests. Step 1308may move the IP address of the existing FSVM 1203 to the available FSVM1209 so that client connections, such as SMB client connections, areretained, e.g., remain open and usable. If the existing FSVM 1203 andavailable FSVM 1209 are on different host machines 1201, 1207, the IPaddress of the existing FSVM 1203 on its host machine 1201 may be movedto the available FSVM 1209 on its host machine 1207. As part of step1306, the FSVMs 1209 of the new VFS may join a domain (e.g., an ACTIVEDIRECTORY domain) associated with the new VFS, and start serving newshares 1210 (e.g., requests to map shares and access those mappedshares). Existing in-use (e.g., open or recently accessed) shares 1204may still be served by the existing VFS 1202.

Step 1310 may incorporate the available FSVM 1209 into one of the newVFSs 1208, 1212, 1216. As an example, each new VFS may be created byproviding the available FSVMs 1209 to the new VFS when it is created. Asanother example, the available FSVMs 1209 may be added to the new VFS asthey are created, and when all the FSVMs 1209 for the new VFS have beenadded, the new VFS may be activated. The particular new VFS into whichthe available FSVM 1209 is incorporated may be determined by anFSVM-to-VFS mapping, which may be provided by a system administrator orgenerated automatically using appropriate load-balancing techniques.

Step 1312 may transition existing in-use shares 1204 (if any) fromexisting FSVMs 1203 to available FSVMs 1209 to which the shares areassigned. This storage transition may be triggered when step 1312 isexecuted, or at other times, e.g., when a system administrator requeststo move the existing in-use (e.g., open or recently accessed) sharesfrom the existing VFS 1202 to the new VFS. When all existing in-useshares served by the existing VFS 1202 have been moved to at least oneof the new VFSs, the existing VFS 1202, including any remaining existingFSVMs 1203 of the existing VFS, may be shut down.

Step 1314 may remove the existing FSVMs 1203 from the existing VFS 1202.The existing FSVMs 1203 may be removed from the existing VFS 1202 one byone. For example, step 1314 may wait for each individual removaloperation for an existing FSVM 1203 to complete before initiatingremoval of the next existing FSVM 1203. Step 1314 may also release anyremaining resources held by the existing FSVM 1203 and provide thoseresources to the corresponding available FSVM 1209. The method 1300 maybe repeated to split other FSVMs 1203 from the existing VFS 1202 andre-created in new VFSs 1209. If each new VFS 1208, 1212, 1216 has notyet been constructed, the method 1300 may construct each new VFS 1208,1212, 1216 after the FSVMs 1209 for the new VFS have been added to thenew VFS.

In particular embodiments, a VFS merge operation may be performed tomerge two or more VFSs 1208, 1212, 1216 together, e.g., upon a systemadministrator's request. The merge operation may form a merged VFS 1208(e.g., as shown in FIG. 12, on host machines 1219 a and 1219 b). Tomerge multiple VFSs, an election may be triggered between the multipleVFSs. The election may be based on characteristics of the VFSs to bemerged, such as virtual IP addresses or preference policies associatedwith the VFSs to be merged, for example. A VFS 1208 that wins theelection is treated as a master VFS. For example, if VFS 1208 wins theelection, then VFS 1208 becomes the master VFS 1208. The other VFSs1212, 1216 to be merged, which are referred to herein as slave VFSs, mayjoin the master VFS's ACTIVE DIRECTORY domain. Available FSVMs 1223,e.g., lightly-loaded FSVMs, may be added to the master VFS 1208, andshares 1210 b,c located at the FSVMs 1209 b,c of the slave VFSs 1212,1216 may be provided to the available FSVMs. The resources of theexisting FSVMs 1209, such as the shares 1210, may be provided to one ormore of the available FSVMs 1223 identified according to a mapping ofresources that are located at existing FSVMs to available FSVMs. Forexample, the mapping of resources may specify that the \HR share 1210 alocated at FSVM 1209 a and the \Development share 1210 c located at FSVM1209 c are to be provided to and located at the available FSVM 1223 a,and the \Finance share 1210 b located at FSVM 1209 b is to be providedto and located at the available FSVM 1223 b. Thus, the shares 1210 b,cfrom the slave VFSs 1212, 1216 may be made accessible as shares 1224 a,bon the master VFS 1208. A snapshot operation may be used to provide thedata located at the FSVMs 1209 b,c of the slave VFSs 1212, 1216, such asthe data accessible via the shares 1210 b,c, to the available FSVMs 1223that are incorporated into the master VFS 1208. Since the snapshotoperation does not copy the data, the VFS merge operation may beperformed without affecting the data stored by the VFSs 1208, 1212,1216, and with zero to minimal down time.

FIG. 14 illustrates an example method 1400 for merging one or moreexisting virtualized file servers (VFSs) to form a merged virtualizedfile server 1208. The method 1400 may be performed primarily by, forexample, a deployment server, a host machine 1207 of a virtualized fileserver, or other component associated with a virtualized file server.The method 1400 may be used to merge existing VFSs 1208, 1212, 1216 inresponse to, for example, a system administrator's request. The method1400 begins at step 1402 by identifying one or more VFSs 1208, 1212,1216 to be merged. Step 1404 may initiate a leader election to elect oneof the identified VFSs 1208, 1212, 1216 as a master VFS. In thisexample, the VFS 1208 (named fs1) wins the election and becomes themaster VFS 1208, although any one of the VFSs 1208, 1212, 1216 may winthe election and become the master VFS in other examples. The otheridentified VFSs 1212, 1216 may then be merged into the master VFS 1208.In one example, the master VFS 1208 may be the VFS that has the highestIP address value. The election may be performed, for example, by causinga distributed lock to be requested for each existing VFS 1208, 1212,1216, where each VFS is represented by a queue entry having anassociated IP address value, and the queue entries are sorted in orderof decreasing IP address value. As an example, the first VFS 1208 forwhich the lock is acquired may be the VFS having the highest IP addressat the time of the election and may be selected as the winner of theelection.

In particular embodiments, once the master VFS 1208 has been elected,clients may connect to it. The identified VFSs 1212, 1216 other than themaster are referred to as slave VFSs. Step 1406 may, for each slave VFS,provide the slave VFS's storage to an available FSVM 1223 of the masterVFS 1208. The available FSVM 1223 may be, e.g., a lightly-loaded FSVM,which may be in the master VFS 1208 prior to step 1406, or may be addedto the master VFS at step 1406. The resources of the existing FSVMs maybe provided to the available FSVMs according to a mapping of resourcesthat are located at existing FSVMs to available FSVMs. A snapshotoperation may be used to provide the data located at each FSVM 1209 b,cin the slave VFSs 1212, 1216 to the available FSVMs 1223 withoutcopying. Step 1408 may re-direct requests to access storage items, suchas shares 1210 b,c, served by the existing FSVMs 1209 b,c of theexisting VFSs 1212, 1216 to the merged VFS 1208's available FSVMs 1223a,b, on which the data may become available to clients as shares 1224a,b. For example, the \HR share 1210 a from VFS 1208 and the\Development share 1210 c from VFS 1216 may be made available on FSVM1223 a (as shares \Admin\HR and \Development 1224 a). Further, the\Finance share 1210 b located at FSVM 1209 b of VFS 1212 may be madeavailable on FSVM 1223 b (as the share \Admin\Finance 1224 b) of themerged VFS 1208. Step 1408 may also move the IP address of each existingFSVM 1209 of the slave VFS to refer to the available FSVM 1223. At step1410, the available FSVM 1223 may start serving clients (e.g., SMBclients) for new and existing shares 1224 associated with the availableFSVM 1223. Step 1412 may stop the slave VFS 1212, add its existingFSVM(s) 1209 b one by one to the new master VFS 1208, and release theresources of the slave VFS 1212 to the new master VFS 1208.

In particular embodiments, when an existing VFS 1202 is split into twoor more new VFSs 1208, 1212, 1216, each share 1203 hosted by theexisting VFS 1202 may be re-assigned to be located at, e.g., served by,one of the new VFSs. Thus a share 1210 a served by one of the new VFSsis not served by the other new VFSs 1212, 1216. The shares may be, e.g.,SMB shares, NFS exported directories, or the like. There may berestrictions on accessing the new VFSs 1208, 1212, 1216 that preventsome users or client devices from accessing the shares 1210 a that arelocated on the other new VFSs 1212, 1216. Consequently, some shares 1210a may be inaccessible to some users or clients. The access restrictionsmay be, for example, network communication restrictions. That is, accessto a share 1210 b (named \Finance) on an FSVM 1209 b of a VFS 1212(named fs2) may not be possible from a particular client because theclient device and the FSVM 1209 b are on different networks and there isno communication link between the two networks. However, clients that donot have access to VFS 1212 may need to access the \Finance share 1210b. The clients may have been able to access the \Finance share prior tothe VFS 1202 being split into the new VFSs 1208, 1212, 1216, but accessrestrictions such as network communication restrictions or otherrestrictions may prevent the clients from accessing the \Finance sharelocated on the VFS 1212.

In particular embodiments, the VFS 1212 may provide access to shares1210 a, c that are located on other VFSs 1208, 1216 to clients that areotherwise unable to access the shares 1210 a, c. The VFS 1212 mayprovide this access by forwarding or proxying file access requests forthose shares 1210 a, c on other VFSs 1208, 1216 to the FSVMs 1209 a,1209 c on which the shares are located. Although examples describedherein uses shares as the storage items being accessed, any suitabletype of storage item may be used instead of or in addition to shares.Other types of storage items that may be used include files, folders(also referred to as directories), and other types of data objects. Thetechniques disclosed herein may be applied to any type of storage item.A storage item contained in another storage item, such as a file ordirectory contained in another storage item such as a share or adirectory, may also be accessible by clients or users to whom the otherstorage item is accessible. For example, accessibility of a firststorage item may apply recursively to other storage items contained inor associated with the first storage item. There need not be a separatesharding map entry for each storage item that is contained in orotherwise associated with a storage item that has a “shareable=true”entry in the sharding map, or the like. Each contained storage item maybe accessible in the same way as the containing or otherwise associatedstorage item that has an entry in the sharding map indicating theassociated storage item is shareable.

In particular embodiments, a first VFS may forward or proxy requests toaccess files located at a second VFSs to the second VFS. The file accessrequests may be, e.g., network file service operations received by anFSVM of the first VFS via a network to open, read, write, or otherwiseaccess files located at the second VFS. The FSVM or other component ofthe first VFS may forward the requests to the second VFS by sending themvia a network to an FSVM or other component of the second VFS. Thesecond VFS may process the requests and send responses to the first VFS,which may forward or proxy the responses to the client device(s) thatsend the corresponding requests. The processing and communication may beperformed by FSVMs or other components of the VFSs. Alternatively, thesecond VFS may process the requests and send responses directly to theclient device(s) if possible, though lack of network connectivitybetween the second VFS and the client device(s), or other accessrestrictions, may prevent such direct responses.

An administrative command may be provided to tag a share 1210 a as a“shareable share” that may be shared across multiple virtualized fileservers 1208, 1212, 1216. The share 1210 a is named \HR in this example.The administrative command cause a VFS 1208 to set an attribute of theshare 1210 a in a sharding map 360 (shown in FIG. 3C) to indicate thatthe share 1210 a is a shareable share. That sharding map 360 may beaccessible to multiple virtualized file servers 1208, 1212, 1216. As anexample, the virtualized file server 1208 may, in response to anadministrative command to tag a share as a “shareable share,” set anattribute named “shareable” of the \HR share in the sharding map 360 to“true.” Shares that are not tagged as “shareable shares” may have no“shareable” attribute, or may have a “shareable” attribute with thevalue “false.”

An access control list may be used to specify which users or groups arepermitted to access the tagged share. The access control list may beassociated with or stored with the entry for the share in the shardingmap 360. The ability to selectively choose certain shares that may beshared across other VFSs 202 may provide a tight security boundary atthe VFS level, along with collaborative access via two different VFSs.

As an example, if a share named Share1 located on a VFS fs1 has beentagged as a shareable share, e.g., in a sharding map, access requestsfor Share1 that are received by a FSVM of a VFS fs2 may be resolved bylooking up the location of Share1 in a sharding table 360 or the like,and forwarded or proxied to the location of Share1, e.g., fs1, therebyallowing Share1 to be readable/writable from multiple different VFSssuch as fs1 and fs2. Clients may access Share1 on both fs1 (e.g., as\\fs1\Share1) and fs2 (e.g., as \\fs2\Share1). In this way, clients thathave permission to access fs2 but not fs1 may access Share1 as\\fs2\Share1.

As an example, an existing file server fs0 may be split into two newfiles servers fs1 and fs2. The existing file server fs0 originallyhosted a share named Share1. After the server fs0 is split, Share1 islocated on fs1, and fs0 no longer exists. Clients that have access tofs1, such as a user U1 of a client device C1, may access Share1 on fs1,e.g., as \\fs1 \Share1 (optionally subject to other access restrictions,such as share-level or file-level permissions). However, clients that donot have access to fs1, such as a user U2 on a client device C2, may beunable to access Share1, e.g., because there is no network connectivitybetween client device C2 and an FSVM of fs1.

FIG. 15 illustrates an example method 1500 for accessing sharedfilesystems on multiple virtualized file servers. The method 1500 may beperformed primarily by, for example, one or more FSVMs of one or moreVFSs 1208, 1212, 1216. The method 1500 begins at step 1502 when an FSVM1209 b receives a request to access a share or other type of storageitem. The request may be received from a client system via the network140. The FSVM 1209 b is included in a VFS 1212 (fs2). The share accessrequest may identify a storage item, such as a share 1210 a, file, ordirectory, and may include parameters that specify an operation to beperformed on the storage item, such as reading or writing data. Thestorage item may be a \HR share 1210 a, for example.

Step 1504 may determine the location of the requested storage item. Forexample, step 1504 may query a sharding map 360 to determine thelocation (e.g., FSVM) of the share 1210 a. In this example, the \HRshare 1210 a is located on the FSVM 1209 a, so the sharding map 360 hasan entry for the share 1210 a, and the entry associates the share 1210 awith the FSVM 1209 a, e.g., (FSVM11, fs1). Further, the share 1210 a hasbeen tagged as shareable, so the sharding map's entry for the share 1210also has a “shareable” attribute with the value “true.”

At step 1506, the FSVM 1209 b may determine whether the share 1210 a islocated on an FSVM of the VFS that received the access request, e.g.,VFS 1212 (fs2) in this example. If so, the FSVM 1209 b that received therequest, or another FSVM of the VFS 1212 (fs2) may process the accessrequest at step 1508, e.g., by performing the requested file serveraccess operation and sending a result to the client that sent the accessrequest. Otherwise, if at step 1506 the FSVM 1209 b determines that theshare 1210 a is not located on an FSVM of the VFS 1212 that received therequest, as is the case in this example, then at step 1510 the FSVM 1209b may check whether the entry for the share 1210 a in the sharding map360 indicates that the share is designated as being accessible by otherVFSs, e.g., has a “shareable” attribute with the value “true.” If theentry indicates that the share is accessible by other VFSs, then at step1512 the FSVM 1209 b may forward the file access request to the FSVM1209 a at which the share 1210 a is located. The FSVM 1209 a may processthe request and send a response directly to the client that sent therequest. Alternatively, instead of sending a response directly to theclient, the FSVM 1209 a that processes the request may send the responsevia the FSVM 1209 b that received the request, e.g., the FSVM 1209 a maysend the request to the FSVM 1209 b, which may in turn, at step 1516,send the response to the client that sent the request.

If step 1510 determines that the share is not designated as accessibleby other VFSs, e.g., has a “shareable” attribute with the value “false,”then at step 1514 the FSVM 1209 b that received the request may send areply to the client with an error indicating that the requested storageitem is not accessible.

In particular embodiments, a virtualized file system (VFS) disasterrecovery system may create backups of data that is stored in a VFS. Adata backup may be a copy of a VFS's stored data, e.g., copies of allthe files, folders, and metadata stored by the VFS. The data backup maybe, for example, a file-level backup in the form of a set of files andtheir paths, optionally stored in an archive file, or block-level backupin the form of a set of disk blocks and associated data structures. Ifdata loss occurs, e.g., because of a disaster, hardware failure, orother type of failure, then the VFS may recover from the failure byrestoring the VFS data from the data backup, e.g., by creating a new VFSand copying the data from the backup to the new VFS. The disasterrecovery system may also replicate delta changes in the storage layer sothat a data backup may be supplemented over time with more recent databy storing changes to the backup in an incremental manner. The changesmay be stored separately from the data backup, so that the data backupneed not be modified when the changes are stored. When recovering from afailure, the VFS may restore the most recent data backup and apply deltachanges that are more recent that the data backup to the restored data.

Replicating delta changes ordinary involves storing relatively smallamounts of data compared to backing up all the data stored in the VFS,and is ordinarily faster than creating a backup copy of the data.However, over time, the storage space used by the delta changes maybecome substantial. Additional backups may be made by copying the data.The delta changes from prior to the latest backup may then be deleted,since the newer backup is more recent that the delta changes. A databackup and delta changes are referred to herein as a replicatedconfiguration.

The backup and delta changes may be stored at a remote site, which maybe at a different geographic location than the VFS. The disasterrecovery system may recover the data stored in the VFS 202 from a backupstored at the remote site. The data may be recovered by reconstructingthe VFS 202 from a replicated configuration, e.g., from thepreviously-generated data backup and delta changes.

In a production environment, the data stored on a VFS 202 may besecurely protected and restored at a remote location without loss of thedata and metadata and with a defined service level. The service levelmay specify that the data is to be recovered within a defined timeperiod, e.g., within a supported Recovery Point Objective. A RecoveryPoint Objective may be, for example, the maximum amount of time forwhich the VFS may be unavailable without irretrievably losing data.

A custom replication policy may be configured for the VFS 202, and theability may be provided to map the VFS 202's configuration between sitesto provide disaster recovery of virtual file services acrossgeographical locations. Particular embodiments may provide the abilityto protect individual shares or share groups by protecting the volumegroup(s) used for file-services storage, e.g., by adding them to aprotection domain (PD). A protection domain (PD) may specify a backupschedule, e.g., every day, every hour, weekly, monthly, or the like. Aprotection domain may also specify a number of snapshots to be retained,e.g., 2, 5, 10, or the like. The VMs in the protection domain may bebacked up by creating a snapshot and replicated to a remote location(e.g., remote-site-1) by replicating deltas after creating the snapshot.A protection domain may include one or more consistency groups (CGs). Aconsistency group may be understood as a subset of VMs (or applications)in a protection domain. All VMs within a consistency group for thatprotection domain are snapshotted in a crash-consistent manner. Asnapshot creates one snapshot for all VMs in a consistency group.

Users may apply the replication and backup policies on the protectiondomain to configure the Recovery Point Objective, recovery sites(alternate cluster or cloud), and replication constraints such asbandwidth and schedule. Particular shares may be mapped to particularremote sites by the protection domain or by the replication and backuppolicies.

Particular embodiments may take lightweight snapshots and transfer thedelta of the snapshots for the given volume groups to remote sites.Along with file-services share data, particular embodiments may alsotransfer the VFS configuration, e.g. file-server size, compute-unitconfiguration, and metadata, e.g., share ACLs, quotas, and so on.Snapshots may copy metadata or an index at the time they are taken. Themetadata or index may include a block map that maps blocks of thesnapshot to blocks stored on a virtual disk. Because of the relativelysmall quantity of data copied by snapshots, they can be nearlyinstantaneous, have relatively little performance impact, and userelatively small quantities of storage space. Snapshots may beimplemented using redirect-on-write, in which updates to existing dataare redirected to a new location. Thus, none of the existing data insnapshots needs to be copied or moved.

Particular embodiments may also provide a simplified user interface toconfigure mapping of network, DNS-servers, active-directory, and so onbetween remote sites. The user interface may provide one-click restoreof VFS file-services on remote sites. In particular embodiments, the VFSbackup system described herein provides a granular level of protection(share or group of shares) to configure different Recovery PointObjectives per share or per group of shares. System administrators orusers may specify custom replication policies to utilize networkresources effectively for replication. The VFS backup system alsoprovides distribution of share replication across multiple remote sitesand multiple recovery points on multiple remote sites for multi-sitefailures. The granularity of the storage items may be specified by thebackup and replication policies. For example, the granularity may beshare-level to indicate that shares are to be backed up as separateunits with consistency preserved inside each share, orgroup-of-share-level to indicate that defined groups of shares are to bebacked up as units with consistency preserved inside each group.

FIG. 16 illustrates an example method 1600 for recovering data afterfailure of a virtualized file server. The method 1600 may be performedprimarily by, for example, a deployment server. The method 1600 beginsat step 1602 by identifying backup data that comprises data stored onthe virtual disks and VFS configuration information, and the first datais identified in accordance with a backup policy. The backup data may beidentified based on a protection domain associated with the backuppolicy. The data stored on the VFS may include one or more storageitems. The storage items comprise one or more shares, groups of shares,files, or directories. The VFS configuration information may specifyconfigurations of one or more File Server Virtual Machines (FSVMs) ofthe VFS.

Step 1604 may send the backup to one or more remote sites for storage.Internal consistency of the storage items may be preserved in the storedbackup data, and granularity of the storage items is specified by thebackup policy. The granularity may be share-level orgroup-of-share-level. The data may be stored on a plurality of blocks ofvirtual disks of the VFS, and may be sent as a snapshot comprising oneor more of the blocks to the remote sites.

Step 1606 may, in response to detection of one or more changes in thebackup data, send the changes to the remote sites in accordance with areplication policy. The changes in the backup data may include one ormore additions to, updates of, or deletions from the backup data. Thechanges may be sent according to a replication protocol. The backup andchanges may be sent via a network in accordance with a Recovery PointObjective (RPO), and the RPO may be specified by the backup andreplication policies. The remote sites may be identified based on thebackup policy, the replication policy, or the protection domain. Thehost machines may be are in a first cluster, and the remote sites may bein a second cluster separate from the first cluster. The backup data maybe identified and sent in accordance with backup constraints specifiedby the backup policy. The backup constraints may include a snapshotschedule and a bandwidth specification. The changes may be identifiedand sent is in accordance with replication constraints specified by thereplication policy. The replication constraints may include areplication schedule and a bandwidth specification. Step 1608 mayrecover the VFS at the remote site(s) by reconstructing the VFS from thestored backup and changes. Internal consistency of the storage items maybe preserved in the restored backup data located on the remote site(s).

FIG. 6 illustrates an example virtualized file server (VFS) environment600 in which a VFS 202 named “FS1” is deployed across multiple clusters618 according to particular embodiments. Different clusters 618 may beat different geographic locations, e.g., in different buildings, cities,or countries. Particular embodiments may facilitate deploying andmanaging a VFS 202 having networking, compute-unit, and storageresources distributed across multiple clusters from a system managementportal or interface such as a system manager 602. The system manager 602may be, e.g., computer program code that can execute on one or more ofthe host systems 201. FIG. 6 also illustrates fault-tolerantinter-cluster sharding of a share “Share1” 614 across compute units 170and clusters 618. FIG. 6 further illustrates andcluster/site/location-aware quotas within the share 614.

Particular embodiments may create a VFS 202 and distribute computeunits, which may be FSVMs 170, to one or more clusters 618. For example,a portal user interface 640 of the system manager 602 may be used by asystem administrator or user to create the VFS 202. While creating theVFS 202, the system administrator or user may be presented with a listof clusters, from which the administered or user may select one or moreclusters. The compute units (e.g., FSVMs 170), networking (IPaddresses), and storage (containers 622) may be distributed to theselected clusters. In the example of FIG. 6, the user has chosen threeclusters, Cluster 1, Cluster 2, and Cluster 3 from the list. In thisexample, three FSVMs 170 are created on each cluster and included in theVFS 202, for a total of 9 FSVMs 170 across the three clusters 618 a-c.Each cluster hosts a separate container 622, which may provide storageservices to the FSVMs 170, e.g., using volume groups (such as volumegroup 626) that contain disk devices. Each container 622 may store aportion of the file server data. The containers 622 are labeledContainer 1, Container 2, and Container 3 in this example. Thecontainers 622 may be hidden from the administrator or user.

Particular embodiments may create one or more shares 614 and distributethe data stored within the shares 614 across the clusters 618. The datastored within the shares may be distributed to multiple storage units,e.g., containers 622, and multiple compute units, e.g., FSVMs 170, whichmay be distributed across multiple clusters 618. The portal userinterface 640 may be used to create the “Share1” share 614 within theVFS 202. A storage pool of multiple virtual disks (vDisks) isconstructed on the FSVMs 170 on the clusters 618. Each storage pool oneach FSVM 170 may be responsible for a subset of the data stored in theshare 614. The share 614 may be sharded at the top-level directoriesacross FSVMs 170 residing in different clusters. For example, differenttop-level directories may be stored on different clusters, but eachsub-directory of another directory is stored on the same cluster as itsparent directory.

FIG. 17 illustrates an example interaction diagram for serving fileaccess requests in a virtualized file server 202geographically-distributed across clusters 618. A system manager 602identifies the cluster 618 and FSVM 170 on which data is to be locatedwhen a storage item such as a file or directory is created by a client620, and directs the VFS 202 to that FSVM 170. The address of the FSVM170 at which the storage item is located may then be recorded in thesharding map 624. In particular embodiments, when an existing storageitem is read or written by a client, the location of the existingstorage item may be found in the sharding map 624.

In a VFS 202, the processing units (FSVMs 170) and data storage units(containers 622) may be sharded, e.g., partitioned, across clusters 618a-c, and may further be sharded across host machines 201 within eachcluster 618. Initially, as shown in Table 6 below, several existingdirectories, e.g., dir1, dir2, dir3, dir4, and dir5, have been createdon Share16 14 of the FS1 VFS 202. The directories may contain files andother directories (not shown). FSVM1, FSVM2, and FSVM3 170 are locatedon Cluster16 18 a, FSVM4, FSVM5, and FSVM6 170 are located on Cluster2618 b, and FSVM7, FSVM8, and FSVM9 170 are located on Cluster3 618 c. Ofthe directories 628 located on Share16 14, dir1 is located on FSVM1,dir4 is located on FSVM3, dir2 is located on FSVM6, dir3 is located onFSVM7, and dir5 is located on FSVM8. Each FSVM 170 within each cluster614 hosts a storage pool created from a subset of the storage providedby the cluster's container 622. A sharding map 624 is stored in adatabase and initially contains five entries that specify the locations(e.g., cluster and FSVM) of Share1's dir1-dir5, as shown in Table 6.

TABLE 6 Share Storage item Cluster FSVM Share1 dir1 Cluster1 FSVM1Share1 dir2 Cluster2 FSVM6 Share1 dir3 Cluster3 FSVM7 Share1 dir4Cluster1 FSVM3 Share1 dir5 Cluster5 FSVM8

At Step 1702, an automatically-invoked process, e.g., a backgroundprocess, may periodically run on a leader FSVM 170 in each cluster 618to calculate the total amount of filesystem space used for each share614 across the FSVMs 170 in the cluster 618. Other usage statistics maybe collected similarly, e.g., the total amount of filesystem spaceavailable on each cluster 618. Step 1702 may send cluster storagestatistics such as the total amount of filesystem space used for eachshare to the system manager 602 via the network 140.

At step 1704, the system manager 602 may receive the cluster storagestatistics 604 for each FSVM 170 on each cluster 618 from the leaderFSVMs 170. The system manager 602 may store the statistics as fileservice statistics 604, which may be, e.g., a data structure in whichstatistics for one or more VFSs 202 are collected and tracked. The fileservice statistics 604 may include filesystem FS1 statistics 606, whichmay include one or more share statistics records 608 for shares 614served by the FS1 filesystem. Share1 statistics 608 contain statisticsfor Share16 14. The Share1 statistics 606 include cluster spacestatistics 610, indicating the amount of storage space used on eachcluster 618, which may be collected as described below. The file servicestatistics 604 may also include optional FSVM CPU statistics 612,indicating the load average (or other measure of CPU usage) of each FSVM170. The FSVM CPU statistics 612 may be collected for a particularcluster when the cluster has been identified, as described below. Inother embodiments, the FSVM CPU statistics 612 may be collected andstored similarly to the cluster space statistics 610. The system manager602 may store the file service statistics 604 in an entity database,e.g., APACHE CASSANDRA or the like. The system manager 602 may be, e.g.,NUTANIX PRISM CENTRAL, which is a multi-cluster system managerresponsible for managing multiple clusters to provide a single,centralized management interface, including a user interface 640.

Arrows are shown in FIG. 6 to illustrate the sending of usage statisticsto the system manager 602. For example, Cluster16 18 a may send amessage to the system manager 602 indicating that 5.5 terabytes (TB) ofspace is used for Share1 on Cluster1. Cluster2 618 b may send a messageto the system manager 602 indicating that 20 TB of space is used forShare1 on Cluster2. Cluster3 618 c may send a message to the systemmanager 602 indicating that 20 TB of space is used for Share1 onCluster3. The system manager 602 may record the spaced used (or otherappropriate storage usage statistics) for each cluster as cluster spacestatistics 610.

At step 1706, a user of a client 620, e.g., a WINDOWS SAMBA client,submits a request to create a new top-level directory named “dir7” onShare16 14. The new directory's full path is \\FS1\Share1\dir7. As aresult, a SMB request message to create the directory is transmitted atstep 1708.

At step 1710, one of the FSVMs 170 may receive the SMB request messageto create a storage item, e.g., a directory or file, such as the requestto create the dir7 directory that was sent at step 1708. A SAMBAinteraction layer of the VFS 202 may send and receive SMB messages viathe network 140.

At step 1712, the VFS 202 may query a database, such as the sharding map624, to determine whether the storage item, which is the directory dir7in this example, is located at any of the existing FSVMs 170. If thesharding map 624 indicates that storage item is not located at any ofthe existing FSVMs 170, as is the case in this example, then at step1714 the VFS 202 may send a request, e.g., a Remote Procedure Call (RPC)or a message, to the system manager 602 to identify a storage locationof the storage item. The location of the storage item as specified inthe sharding map 624 may include a cluster identifier and an FSVMidentifier, which may be numbers, names, or a combination thereof.Cluster1, for example, may be identified by the number 1, and FSVM1 maybe identified by the name FSVM1 or by a fully-qualified domain name(FQDN), such as fsvm1.domain.com. In this example, the storage item dir7is not located at any of the FSVMs 170, so the request is sent at step1714. Otherwise, if the sharding map 624 indicates that the storage itemis already located at an FSVM 170, then the step 1732 may determine thatthe storage item already exists, and step 1734 may send an errorresponse to the client 620.

At step 1716, the system manager 602 may identify a cluster 618 on whichthe storage item is to be located in response to receiving the requestto identify a storage location for the new storage item. The cluster 618may be identified based on the resource usage statistics 604. Forexample, the system manager may retrieve the per-cluster storage usagestatistics for each share received at step 1704, and choose the clusterthat has the least used space. In the example of FIG. 6, Cluster1 ischosen, since its 5.5 TB of used space is less than Cluster2's usedspace (20 TB) and Cluster3's user space (20 TB). The system manager 602may also provide an option to choose the cluster that has the greatestamount of free fileserver container space. Other criteria, such as theaccess latency and bandwidth of the cluster's storage devices, the typesof the storage devices (e.g., magnetic disk drive or solid-state drive),the frequency at which the storage devices are accessed, and so on, mayalso be used to choose the cluster on which the storage item is to belocated.

At step 1718, the system manager 602 may send a request for computeusage statistics of the cluster identified at step 1716 to one or moreFSVMs 170 of the identified cluster, such as the leader FSVM from whichstatistics were received (for the identified cluster) at step 1704, orto multiple FSVMs of the identified cluster. The request may be for theaverage CPU utilization for the past 24 hours for each FSVM in thecluster. The FSVM CPU utilization statistics may be collected at step1718 from the appropriate cluster when needed and need not be stored ina database, but alternatively may be stored (e.g., cached) in thedatabase for later use without being requested at step 1718 if sodesired. In addition to or as alternatives to CPU utilization, otherstatistics that may be collected include the historical CPU load averagemeasured over defined time periods, and other measures of FSVMutilization, such as the average percent of processing capacity used byan FSVM during a time period. In other embodiments, the FSVM CPUstatistics 612 may be collected similarly to the cluster spacestatistics 610, e.g., by the background process described above.

At step 1720, an FSVM 170 of the identified cluster, e.g., a leaderFSVM, may receive the request for compute usage statistics. At step1722, the leader FSVM may send a response containing the requestedstatistics for FSVMs 170 of the cluster to the system manager 602. EachFSVM 170 in the cluster may send its compute usage statistics to theleader, and the leader may forward the statistics to the system manager602. Alternatively, each FSVM in the cluster may send the requestedcompute usage statistics to the system manager 602.

At step 1724, the system manager 602 may receive the compute usagestatistics for the identified cluster and, based on the compute usagestatistics, identify an FSVM 170 on which the storage item is to belocated. The received compute usage statistics may be stored in memoryor in a database as FSVM CPU utilization statistics 612. The identifiedFSVM 170 may be the least-loaded FSVM 170 located on the cluster 618that was identified at step 1716. The least-loaded FSVM 170 may be, forexample, the FSVM having the lowest CPU utilization, or if multiple CPUutilization metrics are received, the FSVM having the lowest average ofthe CPU utilization metrics. The identified cluster is Cluster16 18 a inthis example, and the least-loaded FSVM 170 on Cluster16 18 a is FSVM2.In particular embodiments, the FSVM 170 on which the storage item (e.g.,dir7) is to be located may be identified based on the average CPUutilization for the past T hours for virtual machines within the clusteridentified at step 1710. The time period T may be, e.g., 1, 6, 12, or 24hours, or other appropriate time period. The virtual machines that areincluded in the CPU utilization measurements may be one or more of theFSVMs 170, User VMs 105, and CVMs 110. Other processes, or all processeson the same host machine 201 as the FSVM 170, may also be included inthe CPU utilization that is evaluated when identifying the FSVM 170.Other factors may be used in the identification of the FSVM, e.g.,network latency and bandwidth of communication between FSVMs 170 on theidentified cluster, the number, type, and speed of CPUs available foruse by the FSVM 170, and so on.

At step 1726, the system manager 602 may send a message specifying theidentified cluster and FSVM to an FSVM 170 of the VFS 202. The messagemay include a tuple <Cluster 1, FSVM2 FQDN>, where FSVM2 FQDN is thefully-qualified domain name of FSVM2 (e.g., fsvm2.domain.com).Alternatively, the non-qualified FSVM name, such as FSVM2 without adomain, may be used instead of the fully-qualified domain name.

At step 1728, the VFS 202 may receive the address of the identified FSVMand create the storage item on the filesystem corresponding to theidentified FSVM 170 in the identified cluster. In this example, dir7 isto be located on FSVM2, so the VFS 202 creates a new directory nameddir7 on the filesystem corresponding to FSVM2. The VFS 202 may recordthe mapping (<Share1, dir7>, Cluster 1, FSVM2) in a database such as thesharding map 624 shown in Table 6, and may return a PATH NOT COVEREDmessage to the client 620. Through DFS referral, the SAMBA client 620may request the path for the directory dir7. The FSVM 170 that receivesthe SAMBA path request may look up dir7 in the sharding map 624, and mayfind and return FSVM2's IP address to the client 620. The client 620 maythen access dir7 on the FSVM2 filesystem by sending SAMBA requests toand receiving responses from FSVM2. At step 1730, the FSVM 170 may senda response indicating that the storage object was successfully createdto the client 620.

In particular embodiments, the filesystem on an FSVM 170 may be composedof vDisks. Since vDisks are distributed across the cluster 618 a, thisarrangement provides uniform sharding of storage within the cluster 618a. This sharding strategy causes the containers 622 and FSVMs 170 ofeach cluster 618 to be used, and achieves uniform sharding of storageunits and compute units across the clusters 618.

Particular embodiments may provide cluster-aware sharding andcluster-aware share-level quotas 616. At the time of share creation, asystem administrator or user may be presented with an option toco-locate the data for the share within certain clusters. This optionmay be useful if the administrator or user wishes to have a set ofshares distributed within a certain geographical boundary, and adifferent set of shares distributed across a different geographicalboundary, in which case the above sharding process may still be used. Atstep 1708 above, only those clusters that were selected while creatingthe share may be made available to be considered for sharding. Thistechnique provides cluster-aware sharding.

In particular embodiments, quotas 616 may be set on a file serverservice 202. The quotas 616 may specify a limit on the amount of datastorage to be used for each share within each cluster. As an example,the quotas 616 may specify that the storage limit for each share is 10TB for Cluster1, 100 TB for Cluster2, and 40 TB for Cluster3. Since thesystem manager 602 stores a per-share, per-cluster storage usage in theshare data 608, it can detect when a cluster-level share quota isreached or exceeded. Depending on the quota policy, the user may bealerted when this cluster-level quota is reached or exceeded, or thesystem manager 602 may notify the FSVM leader within the cluster 618whose quota has been reached via RPC. On receiving this notification,the FSVM leader may make all filesystems for that share across FSVMsread-only to respect the storage quota limit.

In particular embodiments, a virtualized file server (VFS) self-healingsystem may automatically identify data corruption and perform datarecovery operations at multiple levels in the storage hierarchy,including the file level, filesystem level, and storage level. Theself-healing system may identify and recover from data corruptionwithout manual intervention. The self-healing system may detect datacorruption at each of the levels and, in response to detecting datacorruption at a particular level, perform a data recovery operation atthat level. The term “data corruption” as used herein may refer toincorrect or missing data resulting from a hardware or software failure.The storage hierarchy levels at which the self-healing service mayidentify and recover from data corruption include the file level,filesystem level, and storage level. Each level has an associated unitof storage and recovery e.g., a unit of stored data that may beidentified as being corrupt and recovered.

In particular embodiments, to automatically identify data corruption,the self-healing service may perform consistency checks, such ascalculating checksums of data blocks and comparing the calculatedchecksums to expected checksum values associated with or stored in thedata blocks. Data loss may be identified automatically by, for example,checking logs that record data write operations. Data loss andcorruption may also be identified by detecting events that may lead tocorrupted or lost data, such as storage device disconnections, read orwrite error, power failures, and so on.

Particular embodiments may identify corrupted or infected data andrecover a consistent version of the data from a VFS 202. Data may beinfected by a virus or corrupted by a file system or storage systemfailure, for example. If the appropriate recovery level is not detectedand data is recovered at the wrong place, then a valid version of datamay be lost. System administrators need not worry about detecting andrecovering a consistent version of data when the system administratordetects the corruption and infected data and manually recovers the datafrom a file system or from storage system. A distributed self-healingmechanism of the VFS 202 may frequently take snapshots of file systemand storage pools, and may monitor the user data at file system andstorage system levels.

As introduced above, the storage hierarchy levels at which theself-healing service may identify and recover from data corruptioninclude the file level, filesystem level, and storage level. Each levelhas an associated unit of storage and recovery e.g., a unit of storeddata that may be identified as being corrupt and recovered. The filelevel's unit of storage and recovery may be a file or directory. Notethat the term “directory” is used herein to refer to either a folder ora directory, and the terms “folder” and “directory” are usedinterchangeably herein. Files and directories may be associated withmetadata, such as names, paths, types, sizes, and access restrictions.For example, corruption of data stored in a file or metadata associatedwith the file may be detected, and the corrupt file may be recovered.Data or metadata associated with the file and stored at a later point intime than the most recent recoverable version of the file (e.g., themost recent backup or snapshot) may be lost. Filesystem or anti-virus orother internal modules can detect file or directory-level infection orcorruption. The anti-virus module may be, e.g., a virus or malwaredetector. In particular embodiments, the self-healing system may monitorthese events and, upon detecting one or more such events, recover theparticular data associated with the event from the previous file systemsnapshot by overwriting the infected/corrupted files/folders.

The filesystem level's unit of storage and recovery may be a filesystem.A filesystem may be a set of one or more files and directories, andassociated metadata. For example, when corruption of filesystem data ormetadata is detected, the entire filesystem may be recovered. Data ormetadata associated with the filesystem and stored at a later point intime than the most recent recoverable version of the filesystem (e.g.,the most recent backup or snapshot of the filesystem) may be lost. Thusmultiple files that are not included in the most recent recoverableversion may be lost when the filesystem is recovered. In particularembodiments, the self-healing system may monitor the checksum(s) of thefilesystem, and upon detecting any discrepancy on the filesystem, e.g.,upon calculating a checksum that does not match an expected checksumassociated with the filesystem, recover the file system from thefilesystem's latest snapshot.

The storage level's unit of storage and recovery may be a storage unit,such as a volume group, storage pool, or storage device (e.g., a harddrive or solid-state drive), which may contain one or more filesystems.The filesystems, which may include files and directories, may berestored if data corruption is detected at the filesystem level. Thus,data or metadata stored in one or more filesystems and more recent thanthe most recent recoverable version of the storage unit may be lost whenthe filesystem is recovered. In particular embodiments, the self-healingsystem may monitor storage-pool corruption and alerts generated by acluster and detect corresponding data losses and corruption. Upondetecting such data corruption or data loss, the self-healing system mayrecover the storage unit (e.g., storage pool) associated with the dataloss or corruption from the latest snapshot.

In particular embodiments, since VFS compute and storage units may bedistributed across multiple FSVMs 170 on multiple host machines 201, theself-healing system may efficiently monitor the corruption and data lossin a parallel and distributed fashion on all the FSVMs or host machines201 of the VFS and detect and recover the lost data on the correspondingFSVM or host machine 201 on which the data is located without affectingthe overall file server 202. The self-healing system may be implementedas a daemon process or other type of background process that runs oneach host machine 201.

In particular embodiments, backups of one or more of the storagehierarchy levels may be made. If a filesystem is corrupt, the backup ofthat filesystem may be restored to recover the filesystem. When afailure (e.g., data corruption) is detected, the self-healing system mayalert the system administrator or user and recover from a backup (e.g.,from a snapshot of the storage hierarchy level at which the failure isdetected). The filesystem provided by a VFS may be implemented usingmultiple separate filesystems associated with FSVMs 170. Each FSVM maybe associated with a filesystem and storage (e.g., a volume group). EachFSVM may recover the data on its associated filesystem and storage. Aleader FSVM may orchestrate recovery of the FSVMs. For example, if theentire filesystem is corrupt, the master may recover a backup of theprevious filesystem by instructing all FSVMs on which at least a portionof the filesystem is located to recover. A directory containing multiplefiles may be distributed across multiple filesystems, so recovery adirectory may involve orchestrating multiple FSVMs.

FIG. 18 illustrates an example method 1800 for detecting and recoveringfrom data corruption in a virtualized file server. The method 1800 maybe performed primarily by, for example, one or more host machines 201 ofa VFS 202. The method 1800 begins at step 1802 by identifying one ormore corrupt units of stored data at one or more levels of a storagehierarchy associated with the storage devices, wherein the levelscomprise one or more of file level, filesystem level, and storage level.At step 1804, when data corruption is detected, the method 1800 maycause each FSVM 170 on which at least a portion of the unit of storeddata is located to recover the unit of stored data. To identify one ormore corrupt units of stored data, the self-healing system is configuredto monitor the corruption and data loss on all FSVMs of the VFS. Themethod may, for example, send instructions to a leader FSVM of the VFSto monitor the corruption and data loss on all FSVMs. When corruption isdetected, the method may send instructions to the leader FSVM toinstruct each FSVM on which at least a portion of the corrupt data islocated to recover the corresponding portion of the corrupt data.

Particular embodiments may back up data stored in a cluster to an objectstore, which may be in a public or private cloud (e.g., AMAZON WEBSERVICES), or to a low-cost storage medium within the same cluster.Particular embodiments may then retrieve the backed-up data as needed torestore files for the file server. Particular embodiments may provide amethod to backup data stored on a virtualized file server (VFS) 202 to alow-cost storage medium hosted on the same physical infrastructure,e.g., on the host machines 201, as the VFS 202. This physicalinfrastructure may include a virtualized server providing an objectstore interface (such as AMAZON S3) and using low-cost storage mediasuch as Shingled Magnetic Recording (SMR) drives. This particularvirtual machine (VM) may act as a backup server for other VMs running onthe same infrastructure.

Particular embodiments of the backup server may be hosted on the sameinfrastructure as the compute and storage. Particular embodiments of thebackup server may be used for low-cost storage media, such as SMR drivesattached to the same infrastructure. Particular embodiments of thebackup server may provide generic object-store interfaces such as AMAZONS3 or the like. Particular embodiments of the backup server may providethe same level of availability as the other highly-available services(such as FSVMs) that run on the cluster.

FIG. 19 illustrates an example method 1900 for backing up data stored ona virtualized file server to cloud-based storage. The method 1900 may beperformed primarily by, for example, a backup server. The method 1900begins at step 1902 by providing, by an FSVM 170 of the VFS 202, anobject store interface to an object store. At step 1904, the method mayback up cold (e.g., infrequently accessed) data stored in a VFS locatedon a cluster of host machines to an object store. The data may be, e.g.,storage items such as files or directories. The object store may be in apublic cloud (e.g., AMAZON WEB SERVICES), or may be a low-cost storagemedium within the same cluster. At step 1906, the method may retrievethe backed-up data, e.g., volume groups, as needed via the object storeinterface to restore the previously backed-up data for the VFS.

In particular embodiments, the FSVM 170 of the VFS 202 may provide theobject store interface (such as that associated with AMAZON S3) tolow-cost storage media, such as Shingled Magnetic Recording (SMR)drives. This FSVM 170 may act as a backup server for other VMs, e.g.,FSVMs, running on the same infrastructure. In this way, the VFS 202 maystore data on and retrieve data from the low-cost storage media.

Particular embodiments of the backup server may be hosted on the sameinfrastructure as the compute and storage, e.g., the same host machinesas the VFS. Particular embodiments of the backup server may be used forlow-cost storage media, such as SMR drives attached to theinfrastructure. For example, the low-cost storage media may be attachedto one or more FSVMs 170 of the VFS 202 without using an object storeinterface such as AMAZON S3. Particular embodiments of the backup servermay provide generic object-store interfaces such as AMAZON S3 or thelike. Particular embodiments of the backup server may provide the samelevel of availability as the other highly available services (such asthe VFS and the FSVMs that provide VFS services) that run on thecluster.

Particular embodiments may include a cloud service as a storage tier ofa virtualized file server. Storage may have multiple tiers, e.g., ahot-data tier for frequently-accessed data, a cooler-data tier foroccasionally-accessed data, and a cold-data tier forvery-infrequently-accessed data. Each tier may have an associated accesstime, and more-frequently-accessed data may be stored on tiers havingfaster access times than less-frequently-accessed data. Each file orfolder may be associated with a tier. In particular embodiments, anyattribute of a storage object may be associated with a tier. Otherpolicies are possible, e.g., by quality of service, or a particularuser's files may be associated with a tier. The hot-data tier may beassociated with a high-speed storage device, such as a static memorydrive. The cooler-data tier may be associated with a medium-speedstorage device, such as a hard disk drive. The cold-data tier may beassociated with a low-speed storage device, such as an optical drive,which may be located at a different physical site than the VFS's hostmachines 201.

In particular embodiments, the access frequency associated with datastored in a VFS may be determined based on usage statistics.Occasionally-accessed data may be moved to a slower storage tier, suchas offsite backup or optical backup. Particular embodiments may thenretrieve the backed-up volume groups as needed to restore files for thefile server.

FIG. 20 illustrates an example method 2000 for storing virtualized fileserver data in tiers of cloud storage having different accesscharacteristics. The method 2000 may be performed primarily by, forexample, a backup server. The method 2000 begins at step 2002 by, inresponse to a request to access a data item, determining an accessfrequency for the data item. The access frequency may be determined byan FSVM 170 on which the data item is located by, for example, measuringthe time between consecutive accesses (e.g., reads or writes) of thedata item. The frequency may be a frequency measured in the past, may bedetermined based on one or more past frequencies (e.g., as an average ofpast frequencies), or may be determined based on an expected futureaccess frequency associated with the data item or the type of the dataitem. For example, data that is known to be infrequently accessed, suchas a backup of accounting records than are expected to be accessed atmost once a year, may be associated with a low frequency. The lowfrequency may be represented as a particular value, e.g., 3, wheremedium is represented by 2 and high is represented by 1. Alternatively,the low frequency may be a particular frequency value, e.g., 1 accessper week or 1 access per month.

At step 2004, the method may, in response a request to store a dataitem, determine a storage tier at which the data item is to be stored ina cloud storage service. The tier may be determined based on the accessfrequency associated with the data. In particular embodiments, the tierassociated with a data item may be a first tier for data accessed at afirst frequency and a second tier for data accessed at a secondfrequency less than the first frequency. Each tier may be associatedwith a frequency of data access, which may be related to the storageaccess time of the tier. Each data item may be associated with a tier,and the tier may be determined based on frequency of access of the dataitem. The first frequency may be, e.g., less than one hour or less thanone day. The second frequency may be, e.g., greater than one hour andless than one day, or greater than one day and less than one week. Athird tier may be with a third frequency that is less than the secondfrequency. The third frequency may be, e.g., greater than one week, orgreater than one month, or never (e.g., a frequency of zero). The accessfrequency may be based on usage statistics.

At step 2006, the method may store the data item in the determined tierof the cloud storage service. The data item may be, e.g., a file ordirectory. The storage service may be a public cloud (e.g., AMAZON WEBSERVICES), or may be a low-cost storage medium within the same cluster.

At step 2008, the method may retrieve the backed-up data item from thetier as needed, e.g., in response to a request for the data by the VFS.The tier associated with particular data may be stored by the VFS in thesharding table or other database. For example, a data item such as afile associated with a cold tier may be stored on the cold tier, and anindication that the data is stored on the cold-tier may be stored in thesharding table. The stored tier indication may be used in subsequentcalculations to determine the tier. Access frequencies may also bestored in the sharding table or other database with each data item andused in subsequent calculations of the tier. As an example, when astorage item is accessed, if access frequencies of a storage item arefound to have decreased since the storage item was last accessed, thenthe tier may be reduced to the next lower tier (e.g., down to thecold-data tier if the current tier is cooler-data). Similarly, if theaccess frequency has been found to be similar, e.g., within 5% or within10% or similar value, then the tier may remain unchanged. If the accessfrequency have been found to have increased since the storage item waslast access, then the tier associated with the data item may beincreased to the next higher tier (e.g., up to the hot-data tier if thecurrent tier is cooler-data).

In particular embodiments, a virtualized server 202 may include ablock-awareness system to maintain availability of VFS services in caseof failure of a block by deploying FSVMs 170 on different blocks. Ablock 201 may be, for example, a rackable unit, which may include one ormore host machines 201. In one example, a block may be a 2U (two unitheight) chassis, which may include 1, 2, or four host machines 201. Inparticular embodiments, if there are three or more blocks 201 a-c forthe same FSVM 170, e.g., the FSVM 170 is located on one block and backupFSVMs ready to take over for the FSVM 170 are located on two otherblocks, then the FSVM 170 may be block-fault-tolerant. If one FSVM 170 aon a block fails and a backup FSVM 170 b on a different block takes overfor the FSVM 170 a, then data loss does not ordinarily occur. Inparticular embodiments, data located at, e.g., stored on, two or threedifferent blocks may remain available if one of the blocks fails. Thatis, when a block on which an FSVM 170 a is located fails, if a backup ofthe FSVM 170 a is on a different block, then the failure can betolerated.

In particular embodiments, when FSVMs are created, three differentblocks may be chosen for each FSVM 170, and the FSVMs 170 may store datain a block-aware manner. Thus, for example, an FSVM 170 located on ahost machine 201 in a first block may have a first backup FSVM 170located on a second block and a second backup FSVM 170 located on athird block. For example, when a first block fails (e.g., due to powerloss affecting the block), the hypervisor 130 on a second block mayattempt to migrate the VMs from the first block to the second block. Ifthe second block does not have sufficient resources (e.g., storage)available, the VM migration may fail. In particular embodiments, ifthere are not enough resources on the available running host machines inthe second block for the VM migration to succeed, then the VFS HAfeatures may be triggered, and online FSVM(s) 170, such as FSVMs in thesecond block, may take ownership of the volume-group of offline FSVM(s),e.g., as illustrated in FIGS. 3F-3H.

In particular embodiments, the user VMs 105 may be prioritized overFSVMs 170 for migration to remaining hosts during block failure. Thatis, migration of user VMs 105 to remaining host machines may beprioritized over migration of a failed FSVM 170 a to the remaining hostmachines. For example, the user VMs 105 may be migrated withoutdisconnecting clients or applications, e.g., using live migration or thelike. The FSVMs 170 do not necessarily need to be migrated in this way,since the VFS can recover from FSVM failures by re-assigning the storageresources from a failed FSVM 170 a to a remaining FSVM 170 b. Thus theuser VMs 105 may be migrated with higher priority than the FSVMs 170 tominimize potential delay or downtime during which the user VMs 105 maycould be unresponsive if they were to wait for migration of the FSVM 170a. The remaining FSVM 170 b may take over with little delay, so that theVFS services may continue to be available while the failed FSVM 170 isunavailable. Thus, file access operations may continue in the user VMs105 without noticeable delay.

In particular embodiments, if multiple host machines failsimultaneously, the recovery process may re-assign the resources (e.g.,storage resources) of the failed host machines to other available hostmachines.

FIG. 21 illustrates an example method 2100 for recovering from blockfailures in a virtualized file server. The method 2100 may be performedprimarily by, for example, one or more host machines 201 of the VFS. Themethod 2100 begins at step 2102 by detecting a failure of one of theFSVMs located on a first host machine. The first host machine is locatedwithin a first block of one or more host machines. Step 2104 maydetermine whether all host machines located within the first blockfailed. If so, step 2106 may identify a second host machine locatedwithin the first block. If not all host machines located within thefirst block failed, step 2108 may identify a second host machine locatedwithin a second block of one or more host machines. Step 2110 mayidentify a failover path to a second FSVM located on the second hostmachine. At Step 2112 may, for each storage resource associated with thefirst FSVM, migrate the storage resource to the second FSVM. Failoversteps are described in further detail with reference to FIGS. 3F-3H.

In particular embodiments, the identified failover path may be one of aplurality of failover paths defined for use when a FSVM fails within thecluster. The identified failover path may be selected by a failoverconfiguration. The failover configuration may select the identifiedfailover path as a previous primary path, a preferred path, or a pathselected based on a round-robin ordering. The failover configuration mayselect the failover path having the least number of outstanding orpending requests as compared to other failover paths. The failoverconfiguration may select the failover path based on determining whetherthe first host machine failed, or determining whether all host machineslocated within the first block failed. The second FSVM may take over anIP address associated with the first FSVM to cause the second FSVM toreceive and process storage requests sent to the first FSVM. To detectthe failure, a leader FSVM, which may be one of the FSVMs, may beconfigured to detect a timeout of communications with the first FSVM ora lack of response from the first FSVM to a periodic status checkmessage.

Particular embodiments may recover from multi-node file service failuresin a virtualized file server (VFS) 201 with minimal down time.Traditional file server deployments protected against single hostmachine failures by having a standby host machine. Detection of servicefailures is not spontaneous and issues can occur with keeping backuphost machines synchronized. Further, if the new active host machine isalso down, there may be no way to recover the service. These issues notonly cause service interruption but may also create complications ifthere are multiple host machine failures.

In particular embodiments, failures of multiple host machines 201 andFSVMs 170 may be detected and recovered from without interruptingservice to clients. Simultaneous failure of up to half of the hostmachines 201 may be tolerated. A distributed cluster health service maybe used to detect failures. The cluster health service or the like,e.g., FSVMs 170 monitoring each other, may detect communicationtimeouts, I/O alerts, or other events that potentially indicate a hostmachine 201 has failed. When a service failure is detected, the otheractive host machines 201 may take over the failed host machine'sservices (both storage and network). Each host machine 201 in a clustermay act as a potential backup host machine, so multiple simultaneoushost machine failures may be tolerated as long as no more than half ofthe host machines 201 in the cluster fail. So, even if the new activehost machine is down, other host machines in the cluster can take overthe new active host machine's storage resources and provide continuousavailability. Thus, clients of the VFS 202 do not ordinarily perceiveany downtime or service interruptions when multiple host machines in thecluster are down.

As an example, when an FSVM 170 named FSVM-1 that owns two volume groupsnamed 1A and 2A fails, another FSVM-2 may become the owner of the twovolume groups 1A and 2A and provide file services for storage items thatare stored on those two volume groups. Suppose another FSVM, e.g.,FSVM-4, fails prior to FSVM-1 's recovery, so there are multiple failednodes at the same time. FSVM-4 has two volume groups 1B and 2B. WhenFSVM-4 goes down, another free FSVM, e.g., FSVM-5, may take over forvolume groups 1C and 2C and provide file services for storage items thatare stored on those two volume groups. In particular embodiments,another free FSVM is available as long as less than half of the FSVMs inthe cluster have failed.

In particular embodiments, for each of the failed FSVMs, the healthmonitoring service may select the live FSVM to which its correspondingIP address is assigned based on affinity. Affinity may pertain to anaffinity of requests served by the failed FSVM with: a configuration ofthe live FSVM, a protocol to which the requests conform, resourcesassigned to the host machine on which the live FSVM is running; orsecurity restrictions applying to storage resources previously served bythe failed FSVM.

In particular embodiments, for each of the failed FSVMs, the healthmonitoring service may select the live FSVM to which its correspondingIP address is assigned based on a plurality of failover paths definedfor use when a FSVM fails within the cluster. The failover paths may bedefined as part of an overarching failover configuration.

In particular embodiments, for each of the failed FSVMs, the healthmonitoring service may select the live FSVM to which its correspondingIP address is assigned based on a previous primary failover path, apreferred failover path, or a failover path selected based on around-robin ordering, or any other appropriate ordering/rankingalgorithm.

In particular embodiments, for each of the failed FSVMs, the healthmonitoring service may select the live FSVM to which its correspondingIP address is assigned based on determining which of the live FSVMs hasfew outstanding or pending requests as compared to FSVMs associated withother failover paths.

In particular embodiments, for each of the failed FSVMs, the healthmonitoring service may select the live FSVM to which its correspondingIP address is assigned based on whether a host machine on which theselected live FSVM is running has low resource usage statistics ascompared with other failover paths.

FIG. 22 illustrates an example method 2200 for recovering frommulti-node file service failures in a virtualized file server (VFS) 201.The method 2200 may be performed primarily by, for example, a VFS. Themethod 2100 begins at step 2202, in which a distributed healthmonitoring service may monitor VFS FSVMs 170, e.g., FSVMs that areincluded in a VFS, to determine whether any of the FSVMs has failed oris having problems. A failed FSVM may be, for example, an FSVM on a hostmachine that has crashed or is not operating correctly. As anotherexample, a failed FSVM may be an FSVM that is not operating correctly,e.g., by being unresponsive to communication via the network 140, or bysending incorrect responses, such as responses that do not conform to acommunication protocol to which the FSVM is expected to conform. At step2204, the distributed health monitoring service may detect that aplurality of the VFS FSVMs have failed, and that the failed FSVMs form aportion of the FSVMs comprising the VFS. Step 2206 may, for each of thefailed FSVMs, select a live FSVM to take over for the failed FSVM. Thelive FSVM may be, for example, an FSVM that is operating correctly,e.g., a failure-free FSVM. Step 2208 may, for each of the failed FSVMs,reassign an IP address corresponding to the failed FSVM to a live one ofa plurality of live ones of the FSVMs.

In particular embodiments, a VFS 202 may provide individual users withstorage system information about the file server that is specific to theuser, based on configuration information of VFS 202. For example, VFS202 may provide a user with information about the portion of a share ordirectory that they are permitted to use for data storage and retrieval.VFS 202 may also provide the user with information that is customizedaccording to a user role that is assigned to the user, or according to alevel of quality of service guaranteed to the user.

In particular embodiments, a VFS 202 may provide usage quotas to limitthe amount of file server space that can be used by particular users tostore data. As an example, consider a scenario when many users havetheir home directories on the same share. Existing technologies displaythe user's drive size as being the same as total share size, therebygiving the user the perception that the total share size is available towrite data. However, the user's quota limit may be less than the totalshare size, in which case the total share size is not an accurateindication of the amount of space available to the user to write data.In scenarios such as a user's home directory being accessed forread/write operations, the user may not be able to determine how muchdisk quota is assigned to the user or how much actual space is availableto the user to write data. When the user's quota limit has been met orexceeded, any further writes to the share requested by the user mayfail. Thus, to avoid data loss or other failures that may result fromunexpected exhaustion of storage space, particular embodiments provideinformation about the portion of the share that that user is permittedto use.

Particular embodiments may provide user-specific data to each user, sothat when the user accesses their Virtual Desktop Infrastructure (VDI)environment, e.g., in a user VM 105, information about the user'sstorage usage may be presented. For example, users may view the totalsize of the share using a command or query for the total size of a disk,or as part of the output of another command, such as a command thatcauses a list of files to be presented to the user. However, the totalsize of a disk may be greater than the user's quota limit, as describedabove. Thus, particular embodiments may provide a value related to theuser's quota, such as the quota limit, in response to requests for thesize of a disk in place of the size of the disk. Note that the term diskis used herein to refer to any type of mappable or mountable portion ofa filesystem, such as a share, disk partition, or directory.

As another example, the user may be presented with their home directoryas a mounted drive, and with data specific to their disk portion, suchas the quota limit, disk capacity, average rate of utilization of space,frequency of disk accesses, file type, and so on. When a soft quotalimit is reached, e.g., an average past disk space utilization quota,the user may be alerted through email or other form of communicationthat their disk quota is about to be or has been exhausted. Inparticular embodiments, when a soft quota limit is reached,less-frequently-accessed files, folders, and other items may be archivedautomatically to the cloud or other storage.

In particular embodiments, to implement the replacement of the diskcapacity with the quota limit as described above, a network filesystemrequest for the amount of free space on a disk, a WINDOWS SMB request,may be intercepted, e.g., by an FSVM 170. A response to the request maybe sent based on the properties of the individual user's disk portion,e.g., the user's quota limit, instead of the share's properties.

In particular embodiments, the storage system information requested maycomprise a number of all storage items stored in a specified location,such as a SMB share. The response may be customized to report a numberof storage items associated with the user as the number of all storageitems stored in the specified location. The response may be customizedto label the number of storage items associated with the user as thenumber of all storage items stored in the specified location. The numberof storage items associated with the user may include storage itemswherein the user is identified as an owner of the storage item or theuser belongs to a group of users identified as an owner of the storageitem.

FIG. 23 illustrates an example method 2300 for providing storage systeminformation about a virtualized file server (VFS). The method 2300 maybe performed primarily by, for example, a virtualized file server. Themethod 2300 begins at step 2302 by receiving a user-originated requestfrom a UVM 105 for storage system information, e.g., information about adisk or other mappable or mountable portion of a filesystem, such as ashare, disk partition, or directory. The storage system informationrequested may include a total size of storage available to the user. Theuser may have an associated storage quota limit, and the total size ofstorage available to the user may be limited by the storage quota limit.The request may be received via a network 140 and may be, e.g., an SMBrequest for metadata associated with a disk. In step 2304, the VFS mayretrieve configuration information of the VFS that is specific to theuser, such as the total size of the disk. In step 2306, the VFS maygenerate and send a response via the network, e.g., an SMB response. Theresponse may be customized to report the storage quota limit as thetotal size of storage available to the user. If the total amount ofstorage available on the disk is less than the amount permitted by theuser's quota limit, then the response may report the total amount ofstorage available on the disk. As an example, the user's quota limit forthe disk may be included in the response in place of the value of thetotal size of the disk, though the quota limit may be labeled as thetotal size of the disk in the response. In step 2308, the UVM mayreceive the response. In step 2310, the UVM may present the storagesystem information (e.g., the user's storage quota limit) to the userinstead of the total size of the disk. The quota limit may be presentedas output of a command that the user invoked to get information aboutthe disk, or as output in a window in a graphical user interface thatdisplays information about the disk. The quota limit may be labeled as atotal size of the disk in the output.

Particular embodiments may provide high availability of storage servicesin a VFS. In traditional file server deployments, high-availability issupported by configuring host machines as pairs where the storageresources are inter-connected between two host machines. So, if one ofthe host machines fails, the other host machine in the pair may takeover the storage resources along with the IP address. One limitationwith this approach is that an even number of host machines is needed inthe cluster.

In a VFS 202, minimal to zero disruption is desired in case of afailure. In the VFS 202, each host machine 201 a in a cluster maymonitor the health of every other host machine 201 b,c in the cluster.In particular embodiments, the monitoring may be performed primarily byFSVMs 170 running on the host machines 201, and each host machine 201(e.g., an FSVM 170 on each host machine 201) may monitor the other hostmachines 201 (e.g., by monitoring FSVMs 170 on the other host machines201). If one of the host machines 201 experiences down time because ofeither a planned shutdown or unplanned host machine failures, anotherone of the host machines 201 may become a leader (e.g., by acquiring alock associated with the storage using APACHE ZOOKEEPER or the like),and may begin taking over the storage resources of the failed hostmachine 201. Note that the host machine 201 that experiences down timemay be referred to herein as a failed host machine 201 for simplicity ofdescription, even though the host machine 201 may be down (e.g.,unavailable) for reasons other than a hardware or software failure.

In particular embodiments, when one of the host machines 201 fails, eachof the non-failed host machines may have an equal chance of becoming theleader. For example, if one of the FSVMs 170 a in a VFS 202 fails, theneach of the other FSVMs 170 b,c in the VFS 202 may have an equal chanceof becoming the leader FSVM 170. When a host machine 201 fails, the IPaddress of the failed host machine fails over to the new host machine(or FSVM 170) that takes over for the failed host machine, so thatclients can continue to contact the new host machine (or FSVM) withoutdisruptions. There may be a queue for each type of resource, e.g., aqueue for host machines, a queue for vDisks, and so on. Each hostmachine 201 that is up (e.g., in failure-free operation) may be in thequeue for each resource. When a host machine 201 fails or is otherwisedown, the first host machine 201 in the queue for each resource held bythe failed machine is granted a lock on the resource and becomes aleader for the resource. The leader may then provide the resource whilethe failed host machine is unavailable. When the failed host machineagain becomes available, it may again become the leader for theresources for which it was the leader prior to its failure.

To improve load balancing, the failover storage resources may bedistributed to multiple host machines. The failed host machine'sresources may then be distributed across different host machines. As anexample, 10 storage resources are held by a host machine 201 that fails,and there are ten other host machines 201, each of which has at leastone available resource, then one resource on each host machine 201 maybe allocated to replace the failed host machine 201. Preference may begiven to local storage resources, so that if a host machine is up, thelocal storage of that host machine is preferred for FSVMs 170 on thehost machine 201 over storage of other host machines 201. If multiplehost machines 201 fail, e.g., two host machines 201 fail at the sametime, and there are two backup host machines available, then the twobackup host machines 201 may take over the storage resources of the twofailed host machines.

FIG. 24 illustrates an example method 2400 for providing highavailability of storage services in a VFS. The method 2400 may beperformed primarily by, for example, a virtualized file server. Themethod 2400 begins at step 2402 by monitoring the FSVMs of a VFS todetermine whether any of the FSVMs has failed or is having problems. Instep 2404, the VFS may detect a failure of a first one of the FSVMs on afirst one of the host machines. In step 2406, the VFS may, for eachresource lock associated with the first FSVM, reassign the resource lockto another one of the FSVMs. In step 2408, the VFS may, for each of thereassigned resource locks, select the another one of the FSVMs based ona selected one of a plurality of failover paths defined for use when aFSVM fails within the cluster.

Herein, “or” is inclusive and not exclusive, unless expressly indicatedotherwise or indicated otherwise by context. Therefore, herein, “A or B”means “A, B, or both,” unless expressly indicated otherwise or indicatedotherwise by context. Moreover, “and” is both joint and several, unlessexpressly indicated otherwise or indicated otherwise by context.Therefore, herein, “A and B” means “A and B, jointly or severally,”unless expressly indicated otherwise or indicated otherwise by context.

The scope of this disclosure encompasses all changes, substitutions,variations, alterations, and modifications to the example embodimentsdescribed or illustrated herein that a person having ordinary skill inthe art would comprehend. The scope of this disclosure is not limited tothe example embodiments described or illustrated herein. Moreover,although this disclosure describes and illustrates respectiveembodiments herein as including particular components, elements,feature, functions, operations, or steps, any of these embodiments mayinclude any combination or permutation of any of the components,elements, features, functions, operations, or steps described orillustrated anywhere herein that a person having ordinary skill in theart would comprehend. Furthermore, reference in the appended claims toan apparatus or system or a component of an apparatus or system beingadapted to, arranged to, capable of, configured to, enabled to, operableto, or operative to perform a particular function encompasses thatapparatus, system, component, whether or not it or that particularfunction is activated, turned on, or unlocked, as long as thatapparatus, system, or component is so adapted, arranged, capable,configured, enabled, operable, or operative.

What is claimed is:
 1. A system comprising: a host machine; a storagepool, the storage pool including local storage provided by one or morelocal storage devices connected to the host machine; a file servervirtual machine (FSVM) configured to execute on the host machine, theFSVM configured to form a cluster with at least one other FSVM onanother host machine, the FSVM configured to process an access requestfrom a client to access one or more storage items stored in the storagepool, and a virtualized file server backup system configured to identifya protection domain including a share stored in the storage pool, thevirtualized file server backup system further configured to specify abackup policy for the protection domain.
 2. The system of claim 1,wherein the protection domain further comprises the FSVM.
 3. The systemof claim 1, wherein the protection domain includes volume groupsutilized for file services storage.
 4. The system of claim 1, whereinthe backup policy specifies a share-level granularity for backup.
 5. Thesystem of claim 1, wherein the backup policy includes a mapping of theshare to a remote site.
 6. The system of claim 1, wherein the backuppolicy is configured to preserve consistency of the share.
 7. The systemof claim 1, wherein the FSVM is configured to process an access requestfrom the client to access the share.
 8. The system of claim 1, whereinthe FSVM is configured to process the access request in accordance witha filesystem protocol.
 9. At least one non-transitory computer readablemedium encoded with instructions which, when executed, cause a systemto: add a share to a protection domain, wherein the share is stored in astorage pool configured to be accessed by a cluster of host machines,wherein each host machine of the cluster of host machines includes afile server virtual machine (FSVM) configured to process an accessrequest from a client to access the storage pool, and wherein thestorage pool includes a local storage device provided by at least one ofthe host machines; apply a backup policy for the protection domain,wherein the backup policy includes a policy applicable to the share inthe protection domain.
 10. The at least one non-transitory computerreadable medium of claim 9, wherein the instructions, when executed,further cause the system to: provide a map for the share to a remotesite.
 11. The at least one non-transitory computer readable medium ofclaim 9, wherein the instructions, when executed, further cause thesystem to: add at least one of the file server virtual machines to theprotection domain.
 12. The at least one non-transitory computer readablemedium of claim 9, wherein the instructions, when executed, furthercause the system to: apply the backup policy including a number ofsnapshots of the share to be retained.
 13. The at least onenon-transitory computer readable medium of claim 9, wherein theinstructions, when executed, further cause the system to: configuremapping of an active directory to a remote site.
 14. The at least onenon-transitory computer readable medium of claim 9, wherein theinstructions, when executed, further cause the system to: specify ashare level of granularity for the backup policy.
 15. The at least onenon-transitory computer readable medium of claim 9, wherein theinstructions, when executed, further cause the system to: provide a userinterface for applying the backup policy.
 16. The at least onenon-transitory computer readable medium of claim 9, wherein theinstructions, when executed, further cause the system to: include aconsistency group in the protection domain.
 17. A method comprising:adding a file share to a protection domain, wherein the file share isstored in a storage pool, and wherein the storage pool is configured tobe accessed by multiple host machines forming a cluster, and wherein thestorage pool includes a local storage device provided by at least one ofthe multiple host machines; and adding configuration information for atleast one file server virtual machine (FSVM) to the protection domain,wherein the FSVM is configured to be hosted by at least one of themultiple host machines, and wherein the FSVM is configured to process anaccess request from a client to access the file share.
 18. The method ofclaim 17, further comprising applying a share-level backup policy forthe protection domain.
 19. The method of claim 17, further comprisingspecifying one or more consistency groups within the protection domain.20. The method of claim 17, further comprising mapping the file share toa remote site.
 21. The method of claim 17, further comprising backing upthe protection domain by creating a snapshot.
 22. The method of claim21, further comprising replicating deltas after creating the snapshot.